JAV database

[Haidon]

I've been following both this thread, and your thread, Casshern. I'm always interested to see how people choose to use coding to help them in their own personal pursuits. I toyed with some way to do something similar myself, but I just ended up using an Excel spreadsheet. Boring, lazy way of doing it. The methods you two are using are definitely more interesting a the very least.

 

[Casshern2]

I've been following both this thread, and your thread, Casshern. I'm always interested to see how people choose to use coding to help them in their own personal pursuits. I toyed with some way to do something similar myself, but I just ended up using an Excel spreadsheet. Boring, lazy way of doing it. The methods you two are using are definitely more interesting a the very least.

@Haidon, there is nothing lazy about keeping up with an Excel spreadsheet (so long as you keep it up! ). The beauty of it is so long as you're not doing anything too fancy in it, it can become your data source later for something else. The approach I'm taking is far less complicated than what @ding73ding seems to be up to. It depends on your skills and how much of them you are willing to use towards your goal, and your time. My skills are as limited as the time I have available to work on things, really, so I haven't been back on it in a bit. My goal, though, is to have something others can use which is another BIG reason to keep it simple. I want others to be able to take and do something different with it if they want and not rely on me for much at all.

 

[ding73ding]

I can't quite keep up my promise NOT to work on JAVDB, especially NOT to make a GUI. My lame excuse is that I decided to create a GUI for my work project and it's actually a good idea to build 2 different GUIs for 2 different projects at the same time, for the first time. (that would enforce to learn and implement each framework and widgets honestly and robustly)

The last time I worked on a GUI (not even needing to start from scratch) left me with loath and dread. Happy to learn that things are so much easier now, even from scratch. The JAVDB GUI (ver 0.0.0 pre-demo-prototype) contain only 300 lines of code, more than half of which is shared with my work project (and future projects) without modifying even one line.

Anyway... so far the GUI is only good for browsing the DB. Scrapping, editing, searching, filtering all have not been added to the GUI yet (despite the menu bar showing them). Needless to say there are tons of pending improvement, some are very minor (cleaning up the Python coding for actor and genre, adding a highlight shading to the current vid (SSNI-124 in the screen shot)), some are semi-minor (hooking up menu items and buttons to existing backend functions as mentioned above). Some are very major investment involving creating new backend functionalities beyond my last few posts (Japanese to English mapping (pseudo-translation) for at least actor names and genre, adding a button to launch the video player if the vid is in my collection).

(still... in this case "major" means only more than 24 hours of coding)

Performance-wise I'm very pleased with it, with 770 vids in the database (neglect the 16k database for now, without search/filter GUI, linear browsing is hopeless, the larger database is still usable in the command console) and 400+ cover images, browsing is instantaneous on my i7 workstation. Even on my dinky Celeron 1.83GHz home PC, it's barely noticeable (subjectively... average 0.2 sec per vid). That's taking the easiest and dumbest way of loading images without any pre-fetching and caching. Second screen shows when the cover image is missing (I have no interest to grab all covers since I don't even collect 1% of all JAVs) .

And NOW once again I promise myself to stop working on it, and refocus on getting rich.

 

[Casshern2]

@ding73ding this looks nice! The slider goes through the titles 5 at a time? Do you have a way of recording it in use you can share?

 

[ding73ding]

Hmmm... you are assuming a lot. I already told you it's only good for browsing records in my own limited database and there's no other functionality, what would you see from a recording?

But... I admit I have hooked up a few more functionalities... mainly scrapping data from website using a few different ways to choose what to scrap (e.g. type in a single video code, or scan a folder for media files)

The UI is extremely basic and not so user-friendly. The sliding bar is kinda fun to use but turn out not very useful. Instead of mouse drag, I've added binding the arrow keys to scroll the database, left, right jump by 5, up, down jump by 1.

Also showing 5 vids at a time isn't as useful as I thought... oh well as least I wrote that module/widget for a real project.

Sorry the phone keeps "breathing" while recording the vid. But the app isn't good enough to justify proper recording.

 

[ding73ding]

Last vid I had to cut so much to keep file size uploadable. So this vid shows the menu bar (some of the functions aren't working yet) and scrapping data plus cover image by entering a video code.

Also I've tested launching a video player from this app, but to make it really work I have to write functions to scan my network drives for all the vids. Not hard to do... but I shouldn't do it.

 

[Casshern2]

@ding73ding thanks so much for indulging me! I was curious how this looked when working for you. I've used sliders before back in the day but they lost their novelty pretty fast, especially when going through tons of data with no space for the control, it passed things way to easily. Arrow keys are better, you're right.

 

[ding73ding]

Ah... not even arrow keys... My concern was never about user friendly. You might notice I've inflated the (small) db to over 1100 vids. I plan to keep adding every new release, going forward, to the db, but not necessarily go backward to add the historic (pre 2017) vids.

Anyway 1000 is already pass the size threshold for meaningful browsing using whatever UI. The only workable solution is searching (a minor job, which I promise not to do but may end up doing) and cross-referencing (a major job which I really won't do).

Ahh... right... not quite true. Browsing (by arrow keys) is pretty useful for checking out the latest releases from past few days. About 10-50 vids at a time. Way way back I actually wrote several programs (at diff times) that let me press a number key (not even pressing <Enter> or click [OK]) to rate a vid and auto-advance to next vid. Fastest way to judge a vid by its cover, en mass.

But this is the first time I'm keeping data of all vids (not just my own collection), so it's perhaps especially important to have that fast rating system so I can un-show the low-rated vids (which is the majority) most of the time.

 

[Casshern2]

Wait, so you're creating this to be an all-inclusive database and not just what you have acquired? Just for your personal use? Why catalog all new releases if you may not grab them all eventually?

 

[ding73ding]

Damn... lost my reply. Oh well teach me to write too long!

Short ver: it turns out to take minimal extra work to grab all JAV data. It lets me perform crazy silly research such as "how does each idol's ratio of solo vs non solo vid changes year to year?"

The key thing to do is to have easy and quick way to filter the database for browsing surfing and digging.

The program has now crossed the threshold of functionality that it's now my main workflow for browsing and keep record of JAV (no more Excel!).

 

[Casshern2]

Wow

 

[ding73ding]

Before you get too impressed. Note that my "program" runs two different way. The GUI is pretty (by my humble standard) and "somewhat" easy to use, but it only perform a few functionalities. For most functionalities that are less than the most frequently repeated, I have not yet (and may never) code them into the GUI. Those functionalities are performed on the command line (first quit the GUI).

Both GUI or commandline version are not very user-friendly. But some frequently used operation can be done very quickly. And certain advanced search/filtering is getting pretty powerful.

 

[Casshern2]

As fate would have it...just this Friday my boss told me "I need you to learn something called Python so you can help with work on our website."

 

[ding73ding]

Awesome! Now there's someone I can trust my project to. (now go buy the book Python Pocket Reference 5Ed by Mark Lutz, 6Ed might be overdue, but you can't wait.)

 

[ding73ding]

So... I have mostly kept my promise to myself: not to waste my life to build killa-app for JAV database but only made small tweaks that both improve the usability and along the way help me learn Python killaskills. Which is to say until now I have nothing to report here.

Then one day I had a lunch time to kill and I decided to really learn OO on Python, and I would learn it by (gingerly) developing a "big" new feature on JAV database: bilingual support for (to start with) performers (in my program, I follow IMDB internal format call them all "actor" even thought I am only tracking actresses now).

So what I created (one lunch time and a long coffee break later) was an independent module that augment my JAV database but can also be added to any Python project. The database is kept as an Excel file and you add and edit entries using Excel (or many free software can also do it), and my module let you easily translate between Japanese and English full names or partial names.

The Excel format is super simple. There are quite a few extra fields for future expansion, for now "EngName" and "JapName" are the required fields. AKAs provides a placeholder for multiple aliases for the same performer.


"avactors" is implemented as a Python class derived from DataFrame (from pandas, Python's standard library for data analysis). So you immediately can use pandas' powerful tools to work with the database without me writing any code. I only added a few E2J and J2E methods to DataFrame. So a little demo:

(in following, all the 'NaN' and 'NaT' are missing data)


So you can supply a full or partial Japanese or English name and get back one or more records (each record holds all the data for that idol), or get one or more names in Japanese or English. You can also just straight up do J2E or E2J translation (provided you have the correctly spelled name).

pandas' features let you slice and dice the data any which way. So for example (assuming you have filled Excel file with such data) it's easy to request all the performers who debuted when her age was over 24.

One more fancy demo, say I kind of forgot my idol's name, only there 'ami' in there somewhere. So I do a partial name with Epart(), but I got 12 results... too many, say I remember she's "ami" but not "nami", so I exclude all the "nami", which give me 9 results (let's pretend I decide to accept 9 as narrow/precise enough). So finally I get those 9 names in Japanese:

 

[keanurefresh]

Do you know a database or website where i can find a male jav actor and what movies he starred in ?

 

[wanderboy7]

I found a website have all jav actress and movies.

http://www.jpavdb.com

 

[R18.com]

I found a website have all jav actress and movies.

http://www.jpavdb.com

If you see the images info you will see that all the images are from DMM so this site just take the information from DMM/R18
This site has 194263 movies listed while DMM/R18 has near 290,000 movies so still missing lots of data.

 

[keanurefresh]

If you see the images info you will see that all the images are from DMM so this site just take the information from DMM/R18
This site has 194263 movies listed while DMM/R18 has near 290,000 movies so still missing lots of data.

Hi, so I'm curious about your JAV database.
Is there any particular reason why your database doesnt include the actors?
Because in the west , they do.

Maybe it is a culture reason or something.
Because I found some similiar database like yours, but all of them doesnt include actors either.

 

[R18.com]

Hi, so I'm curious about your JAV database.
Is there any particular reason why your database doesnt include the actors?
Because in the west , they do.

Maybe it is a culture reason or something.
Because I found some similiar database like yours, but all of them doesnt include actors either.

Hi,
Yes, this is a cultural difference.
In Japan the actors does not have the following and actress has.
In other words, in Japan people do not buy the movies because of the actor but because of the Actress. So the actors are not considered that important.
Also Japanese men may feel jealous if the actor is fit and good looking, that is why most of the movies the actors are average or under average looking.