Author |
Message |
bigB
|
Posted: Wed Oct 08, 2008 10:52 pm |
|
Joined: Mon Feb 06, 2006 5:11 pm
Posts: 353
Location:
Brisbane, Australia
|
Hello Everyone,
I've been working on a video meta data scanner. I know this has been done to death, but i wasn't really happy with what already existed. I have created a google code project for it, so that bugs and requests can be tracked.
http://code.google.com/p/mythvideo-scanner/
If you run the mythvideo-scanner.pl script with no parameters, it will look for them in the mythconverg database.
The main operation of the script is scan a path looking for files that mythvideo supports. It looks for an existing entry in the database, if the plot of that entry is short (less than 20 characters) it will process the file, else it will skip the file. It will then determine title, season, and episode. If the file looks like a tv show, it will look up data on tvrage.com, if it looks like a movie then it uses the imdb.pl script that comes with mythvideo to search imdb. If the tv show search fails, it will fall back to imdb.
The script is designed to work on the nastiest/messiest of video directory structures.
Usage:
./mythvideo-scanner.pl <options> <path>
<path>
Specifying a path is optional, it will by default use the path found in the database. Both directories and files are supported.
Supported options:
--update
Conduct searches for all files found.
--dryrun
Don't modify the database, good to see how well it parses your files.
--getposter
Look for, and download posters. Posters are stored in the same directory as the video. Symbolic links are also created for 2 levels of folder images.
All feedback is welcome.
Installation
These are all done as root user.
Prereq
Code: # pacman -S make subversion setuptools # cpan Text::Levenshtein
Mythvideo-scanner Code: # cd /usr/local/src # wget -O - http://mythvideo-scanner.googlecode.com/files/mythvideo-scanner.2.01.tar.gz | tar xzf - # cd mythvideo-scanner # make upgrade # make install
BigB.
Last edited by bigB on Thu Jul 15, 2010 7:58 pm, edited 5 times in total.
|
|
Top |
|
![](images/spacer.gif) |
Gibble
|
Posted: Sat Oct 11, 2008 9:35 am |
|
Joined: Thu Sep 30, 2004 12:26 pm
Posts: 468
Location:
Canada
|
I got "multiple paths not supported yet", I seem to get this often with mythtv software. Even mythweb doesn't work for videos because of it.
_________________ "The amount of time needed to solve a problem is inversely proportionate to the complexity of the solution" -- Me
KM: R5.5
CPU: Athlon 3800+
Vid: nvidia 7300GT
Snd: Chaintech av-710
Tuner: PVR150
HD: 1000gb sata + 750gb sata + 500gb usb
|
|
Top |
|
![](images/spacer.gif) |
Gibble
|
Posted: Sat Oct 11, 2008 9:41 am |
|
Joined: Thu Sep 30, 2004 12:26 pm
Posts: 468
Location:
Canada
|
I'm not quite sure your TV Rage parsing is perfect though, it worked for my Smallville episodes, but my episodes of Dark Angel returned
/myth/video/USB/Dark Angel/S1 - E10 - Red.avi
Title: S1 E10 Red
Season: 1
Episode: 10
Appears to be a TV episode, looking up TVRage website
TVRage has let us down, looking up imdb.
Yet, they appear in, http://www.tvrage.com/Dark_Angel/episode_list , specifically http://www.tvrage.com/Dark_Angel/episodes/42694/01x10
The data page is good and provides a good link
http://www.tvrage.com/quickinfo.php?sho ... 10&exact=1
http://www.tvrage.com/Dark_Angel/episodes/42694
Not sure what happened yet that it couldn't read it.
There's also a parsing error if the file name contains an apostrophe. And you should also ensure there are no '--' combinations while you're at it.
/myth/video/USB/Dark Angel/S2 - E02 - Bag 'em.avi
DBD::mysql::db selectall_arrayref failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'em.avi'' at line 1 at modules/MythTV.pm line 112.
Title: S2 E02 Bag 'em
Season: 2
Episode: 2
Appears to be a TV episode, looking up TVRage website
TVRage has let us down, looking up imdb.
Lastly, when I ran it with --getposter, it skips files becaue the plot length is long enough, imho, in this case, it should check for the poster, as many files have plots, but no posters on my machine :p
Other than that, it worked quite well. If I get a chance I might take a look at the script and look into some of these things for you.
-c
_________________ "The amount of time needed to solve a problem is inversely proportionate to the complexity of the solution" -- Me
KM: R5.5
CPU: Athlon 3800+
Vid: nvidia 7300GT
Snd: Chaintech av-710
Tuner: PVR150
HD: 1000gb sata + 750gb sata + 500gb usb
Last edited by Gibble on Sat Oct 11, 2008 2:19 pm, edited 1 time in total.
|
|
Top |
|
![](images/spacer.gif) |
Gibble
|
Posted: Sat Oct 11, 2008 2:13 pm |
|
Joined: Thu Sep 30, 2004 12:26 pm
Posts: 468
Location:
Canada
|
One last thing, shouldn't the posters go in the .covers directory?
_________________ "The amount of time needed to solve a problem is inversely proportionate to the complexity of the solution" -- Me
KM: R5.5
CPU: Athlon 3800+
Vid: nvidia 7300GT
Snd: Chaintech av-710
Tuner: PVR150
HD: 1000gb sata + 750gb sata + 500gb usb
|
|
Top |
|
![](images/spacer.gif) |
Gibble
|
Posted: Mon Oct 13, 2008 9:03 am |
|
Joined: Thu Sep 30, 2004 12:26 pm
Posts: 468
Location:
Canada
|
Oh, and I just noticed it doesn't grab the year for most movies.
_________________ "The amount of time needed to solve a problem is inversely proportionate to the complexity of the solution" -- Me
KM: R5.5
CPU: Athlon 3800+
Vid: nvidia 7300GT
Snd: Chaintech av-710
Tuner: PVR150
HD: 1000gb sata + 750gb sata + 500gb usb
|
|
Top |
|
![](images/spacer.gif) |
bigB
|
Posted: Mon Oct 13, 2008 4:48 pm |
|
Joined: Mon Feb 06, 2006 5:11 pm
Posts: 353
Location:
Brisbane, Australia
|
Hi Gibble,
Thanks for trying it out.
Quote: Gibble Oh, and I just noticed it doesn't grab the year for most movies. I found a bug, where the year is not added if the an entry exists in the database. Quote: Gibble One last thing, shouldn't the posters go in the .covers directory? Since i add and delete videos quite frequently, it is easiest to delete the poster file if it is in the current directory. Due to an other project i am working on, i will add a method to place the posters in the location pointed to by the database. Quote: Gibble I got "multiple paths not supported yet" This is just a warning that the script is not compatible with mythvideo's support for multiple paths. I have added a ticket to add support for this. Is it something you need, or just noticed? Quote: Gibble when I ran it with --getposter, it skips files becaue the plot length is long enough, imho, in this case, it should check for the poster, as many files have plots, but no posters on my machine Good point. Quote: Gibble /myth/video/USB/Dark Angel/S1 - E10 - Red.avi Title: S1 E10 Red Season: 1 Episode: 10
The problem my script is having with this naming convention is that the series title is in the directory name (Dark Angel). Admittedly "S1 E10" should not make it into the title, but I'm not sure how i get a script to choose between "Dark Angel" and "Red". One assumption i have made on other parsing techniques, is that anything to the right of the season episode numbers is NOT the series title.
I should be able to make this particular naming convention work.
Thanks for the feedback.
BigB.
|
|
Top |
|
![](images/spacer.gif) |
nharris
|
Posted: Tue Oct 14, 2008 1:52 pm |
|
Joined: Thu Sep 07, 2006 11:20 am
Posts: 389
|
First off, thanks for the script. I'm running MythTV SVN. So my feedback could be bogus, but here goes.
I had to comment out the extension fetch from myth. This was providing only ISO and IMG file extensions. I probably need to setup all the extensions I use in myth.
Code: #get supported extensions my @filetypes = ('.avi','.mpg','.mpeg','.mkv'); my $ext = MythTV::query("SELECT extension FROM videotypes WHERE f_ignore = '0'"); if(@$ext>0) { # @filetypes=(); foreach my $extension (@$ext) { push(@filetypes,$extension->[0]); } } print "Seaching for these extensions: "."@filetypes"."\n"; It does not seem to like the ' character in filenames. They cause SQL query problems. Names like "Bob's great adventure" do not seem work. Here is a sample. Code: /myth/video/Pokemon_Season_1/020 - The Ghost of Maiden's Peak.avi DBD::mysql::db selectall_arrayref failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's Peak.avi'' at line 1 at modules/MythTV.pm line 112. Title: Pokemon Season 1/ Season: Episode: 20 Appears to be non-episode, looking up IMDB
The last is not a bug, but I have a directory named "Pokemon_Season_1" which holds all the kids pokemon shows. The files are named by episode like "001 - Pokemon - I Choose You!.avi" and the script will get the episode but not the season which is in the directory name. See above example. No big deal, I can rename everything to a "Sxx Exx" convention.
|
|
Top |
|
![](images/spacer.gif) |
bigB
|
Posted: Tue Oct 14, 2008 9:56 pm |
|
Joined: Mon Feb 06, 2006 5:11 pm
Posts: 353
Location:
Brisbane, Australia
|
Hi nharris,
Thank you for using the script.
Quote: nharris I have a directory named "Pokemon_Season_1"
I'm trying to support as many different season episode number indicators as possible. The more examples i see the better i can make it. The problem with your method (and i'll fix it) is that it is interpreting 020 as season 0 episode 20, with all leading zeros truncated.
I have a few bugs to fix, this will be an area i look into.
BigB.
|
|
Top |
|
![](images/spacer.gif) |
Gibble
|
Posted: Tue Oct 14, 2008 10:03 pm |
|
Joined: Thu Sep 30, 2004 12:26 pm
Posts: 468
Location:
Canada
|
bigB wrote: Hi Gibble, Thanks for trying it out. Quote: Gibble Oh, and I just noticed it doesn't grab the year for most movies. I found a bug, where the year is not added if the an entry exists in the database. Quote: Gibble One last thing, shouldn't the posters go in the .covers directory? Since i add and delete videos quite frequently, it is easiest to delete the poster file if it is in the current directory. Due to an other project i am working on, i will add a method to place the posters in the location pointed to by the database. Quote: Gibble I got "multiple paths not supported yet" This is just a warning that the script is not compatible with mythvideo's support for multiple paths. I have added a ticket to add support for this. Is it something you need, or just noticed? Quote: Gibble when I ran it with --getposter, it skips files becaue the plot length is long enough, imho, in this case, it should check for the poster, as many files have plots, but no posters on my machine Good point. Quote: Gibble /myth/video/USB/Dark Angel/S1 - E10 - Red.avi Title: S1 E10 Red Season: 1 Episode: 10 The problem my script is having with this naming convention is that the series title is in the directory name (Dark Angel). Admittedly "S1 E10" should not make it into the title, but I'm not sure how i get a script to choose between "Dark Angel" and "Red". One assumption i have made on other parsing techniques, is that anything to the right of the season episode numbers is NOT the series title. I should be able to make this particular naming convention work. Thanks for the feedback. BigB.
The multiple directories isn't something I really need, yet :p It's no big deal.
It's looking good.
_________________ "The amount of time needed to solve a problem is inversely proportionate to the complexity of the solution" -- Me
KM: R5.5
CPU: Athlon 3800+
Vid: nvidia 7300GT
Snd: Chaintech av-710
Tuner: PVR150
HD: 1000gb sata + 750gb sata + 500gb usb
|
|
Top |
|
![](images/spacer.gif) |
Girkers
|
Posted: Thu Oct 16, 2008 8:59 pm |
|
Joined: Tue Mar 22, 2005 9:18 pm
Posts: 1422
Location:
Brisbane, Queensland, Australia
|
From looking at files on the internet, the general convention is E##S##. E.g. Heros S03E04 - I Am Become Death.avi
For your info.
_________________ Girkers
|
|
Top |
|
![](images/spacer.gif) |
bigB
|
Posted: Sun Oct 19, 2008 8:26 pm |
|
Joined: Mon Feb 06, 2006 5:11 pm
Posts: 353
Location:
Brisbane, Australia
|
Hi Girkers,
Yeah, i first check for these "standard" methods of labeling season and episode, but when the standard is not adheared to, i want the script to do its best at working it out.
BigB.
|
|
Top |
|
![](images/spacer.gif) |
UniCav
|
Posted: Tue Oct 28, 2008 12:29 pm |
|
Joined: Tue Oct 28, 2008 12:18 pm
Posts: 60
|
I've been testing BigB's script and chatting with him via gmail and it's really coming along. Seems to be one odd bug with discrepancies between the video manager and the browser. After running mythvideo-scanner the videos will show up in the video manager with complete meta-data retrieved by the script. They show as browesable, but most of the time they won't show up in the video browser. Toggling browesable in the manager doesn't fix it, and furthermore if you change something in the manager (i.e. change Type from blank to TV Episode) when you select Done and return to the list the type will show Unknown instead of what you chose.
I'm thinking a possible permissions problem or some kind of sync problem with the database but it would be great to get ideas from those who really know how the system works.
|
|
Top |
|
![](images/spacer.gif) |
g0dl1ng
|
Posted: Sun Nov 02, 2008 7:41 pm |
|
Joined: Wed Apr 04, 2007 9:03 pm
Posts: 9
|
I tried your script and I'm sorry I only know enough to try your script but I hope that I can still provide a little feed back. When I ran the script it found most of my movies and some of my shows but none of the covers display and none of them had year or rating information(MPAA rating). I tried this on version R5.5. Is there anything that I need to do in order to get the covers to display?
|
|
Top |
|
![](images/spacer.gif) |
bigB
|
Posted: Sun Nov 02, 2008 8:17 pm |
|
Joined: Mon Feb 06, 2006 5:11 pm
Posts: 353
Location:
Brisbane, Australia
|
Hi,
Get the new revision from here, quite a few fixes have been applied. I'll be building a new release shortly. No rating information yet.
Covers are currently downloaded into the same directory as the video, this is the last change i need to make before release 1.5. To get images to download you need to specify --getposter. If you have done all this, and still no success, then change your poster directory to /myth/video instead of /myth/video/.covers.
Thank you for using the script, and a new release will be out soon.
BigB.
|
|
Top |
|
![](images/spacer.gif) |
g0dl1ng
|
Posted: Mon Nov 03, 2008 3:08 pm |
|
Joined: Wed Apr 04, 2007 9:03 pm
Posts: 9
|
I tried your latest script and it found all of my movies and most of my tv shows. Great job. But it still seems to have a problem finding covers for them (I have a few but not nearly as many as they found plot and other info for) and this might just be me not know what I should do but my mythweb doesn't show the updated covers for any of the movies that your script found covers for. I also noticed that the IMDB number isn't loaded when it finds the other info. It still seems to have problems loading a year (that's all I noticed as far as what didn't seem to work completely but over all its working great at finding plot and directors. great idea checking the spelling via google). I really like this project and would like to help if you need any. If nothing else I would be more than happy to test your new versions (from a noob point of view) if you need any more testers.
|
|
Top |
|
![](images/spacer.gif) |