View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 12 posts ] 
Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Mon Jun 10, 2019 6:00 pm 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
I saw in another thread that brfransen was saying we will be changing thew way we get our lineup.
brfransen wrote:
If you are using the "North American (SchedulesDirect.org) (Internal)" grabber press the right arrow and enter the user id an password there. But that grabber is going to be removed for mythtv v31 so it is recommended to use one of the xmltv SD JSON grabbers. See https://www.mythtv.org/wiki/XMLTV#Sched ... rect_Setup


So doing some digging in the setup page I can't figure out how to configure this from the GUI. Is it possible or is the only option in the command line?
If someone more knowledgeable about this than I am could make a step by step guide (preferably with lots of colorful pictures) on how all of us Schedules Direct users will need to get our lineup in the near(ish) future that would be awesome.

Thanks,
MattBatt

_________________
My System


Top
 Profile  
 
PostPosted: Wed Jun 12, 2019 6:42 am 
Offline
Joined: Fri Jul 21, 2006 11:12 pm
Posts: 1194
Location: SC
The MythTV Wiki I linked is the most complete guide I have seen. I think all of the XMLTV grabbers are configured via CLI.


Top
 Profile  
 
PostPosted: Fri Jun 14, 2019 10:33 pm 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
Yikes! That is less user friendly than Data Direct.
Thanks.

_________________
My System


Top
 Profile  
 
PostPosted: Tue Aug 13, 2019 3:43 pm 
Offline
Joined: Sat Jan 06, 2007 7:08 pm
Posts: 125
As I live in the north east, I was patiently waiting for phase 2 of the repack to complete before attempting this change.

This gave me some time to read and reread the instructions.

the instructions are accurate, but only become clearer once you perform the steps.

the one thing that slowed me down was a few mystery nonexistent channels that appeared after switching grabbers, and were not added during the channel rescan i did for the repack a couple weeks ago (they also did not exist before the repack). these channels were not listed in SchedulesDirect. using the channel editor in mythweb, i would delete them. unfortunately, those channels reappeared after running mythfilldatabase.


this part of the instructions helped me get rid of those bogus channels:

Quote:
Run these:

tv_grab_zz_sdjson_sqlite --days 0 --config-file $HOME/.mythtv/SD.xmltv
tv_grab_zz_sdjson_sqlite --manage-lineups --config-file $HOME/.mythtv/SD.xmltv
If you get all channels in the lineup, then select the options that will make current and future channels selected.
If you only get a few of the channels you will need to select the appropriate channels. Select a default for existing and future channels, then use option 5 to select which channels you need. This goes though all channels one at a time. You have to complete this for every channel in the list. You cannot stop partway through.
If you need to do the channel selection again because of errors or channel changes, your previous values are remembered and used as defaults.


i performed the second "if" clause in the instructions and was able to delete the bogus channels. in my case, i had too many channels, not too few. so, it was not intuitive that i needed to perform option 5. after rerunning mythfilldatabase, the channels did not reappear.

in the nit-picking department, one of the new features of this grabber is that you get channel icons displayed in the listing on mythweb and in the program guide in myth. unfortunately, some of the icons are black, and as a result are not visible on the black or dark blue backgrounds. here is one example icon that is not visible:

https://s3.amazonaws.com/schedulesdirect/assets/stationLogos/s28711_h3_aa.png

_________________
DH87MC i7-4770 16GB ram Xonar Essence ST geforce 710 LinHes 8.6


Top
 Profile  
 
PostPosted: Fri Aug 30, 2019 11:15 pm 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
OK the DataDirect mode stopped working recently and apparently ran out of data TODAY!!!
I'm not sure what broke but it no longer liked my username and password.
SO I was forced to update to the new JSON mode and boy did I suck at it 2 hours later and it's finally working. Try this first then move on to command line. https://www.mythtv.org/wiki/XMLTV#MythTV_Setup
1) I hate the new setup GUI. They made it so hard to enter answers.
2) the order of the guide is way out of wack, at least I think. I ended up doing most of the steps 3 times because it didn't work.
3) when you go into Video Source setup and you select the source you want to edit (or create) the "Listings Grabber" you want is at the end of the list called "Schedules Direct JSON API (xmltv)"
4) it seems like using the GUI in Mythtv-Setup is really just going through the same command line steps except that it works (when I finally selected the correct grabber)
5) the guide says that if you use ATSC tuner you will have to manually edit the channel IDs. When you do it though the Mythtv-Setup GUI it worked fine.
6) I might have to spin up another VM to do a step by step walk through.

_________________
My System


Top
 Profile  
 
PostPosted: Thu Dec 12, 2019 8:38 am 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
Can anyone help troubleshoot this? I followed the instructions to setup the sqlite3 grabber, and all seemed to go well. All the channels, channel icons, and 21 days of data were downloaded. However, every day since then mythfilldatabase fails with the same error. I ran it last night, and here is the relevant snippet of the log from that run:

Code:
2019-12-11T13:01:35.536901-05:00 mythfilldatabase[7028]: I CoreContext filldata.cpp:379 (GrabData) XMLTV config file is: /home/mythtv/.mythtv/SD.xmltv
2019-12-11T13:01:37.539270-05:00 mythfilldatabase[7028]: E CoreContext filldata.cpp:441 (GrabData) FillData: XMLTV grabber returned error code 255
2019-12-11T13:01:38.039736-05:00 mythfilldatabase[7028]: E CoreContext xmltvparser.cpp:672 (parseFile) Error in 1:1: unexpected end of file


Clearly the grabber doesn't like something, but I can't find anything on the internet about code 255. I re-ran the grabber setup, and it looked good with the correct lineup and password. Running mythfilldatabase -v in a terminal window didn't show anything helpful. Is there anything else I can check to start getting valid guide data again?

Edit: running the grabber manually now gets this error, though it worked the first time with days=0:

Code:
tv_grab_zz_sdjson_sqlite --days 21 --config-file $HOME/.mythtv/SD.xmltv
WARNING: Perl module JSON::XS not installed.  JSON encode/decode performance will be poor.
Opening the local database
Can't call method "errstr" on an undefined value at /usr/bin/vendor_perl/tv_grab_zz_sdjson_sqlite line 3857

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Thu Dec 12, 2019 8:52 am 
Offline
Joined: Sat Jan 06, 2007 7:08 pm
Posts: 125
i would suggest looking at the file

Code:
/home/mythtv/.mythtv/SD.xmltv


it seems as though the "parse error" message relates to that file

_________________
DH87MC i7-4770 16GB ram Xonar Essence ST geforce 710 LinHes 8.6


Top
 Profile  
 
PostPosted: Thu Dec 12, 2019 10:17 am 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
Thanks. It seems like the error is in the grabber script, specifically line 3857. Here's the surrounding code there:

Code:
$DBH = DBI->connect("DBI:SQLite:dbname=$dbname", "", "",
                        { RaiseError => 0, PrintError => 0, AutoCommit => 0 });

    if (!defined($DBH))
      {
        print (STDERR "Unable to open database file $dbname: " . $DBH->errstr . "\n");
        exit(1);
      }

From what I've been able to piece together, it seems the database is failing to connect to the database in 3852. That is leaving $DBH undefined rather than false. When it gets to 3857 and attempts "$DBH->errstr," it gives the error "Can't call method "errstr" on an undefined value." So, as far as I can tell, the underlying problem is that it isn't connecting to the database. I'm not sure why it connected to the database just fine the first time I ran the setup, but I'll poke through the setup again to see if I can figure out where it's going wrong.

I'll throw out another tidbit I just learned since I've noticed my system practically shuts down during mythfilldatabase:
Code:
When using XMLTV, mythfilldatabase can use a lot of main memory. If you have 200 or more channels and you are processing 21 days of schedules it can use 5 GB of resident memory.
I'm going to be adding the --no-allatonce option to my mythfilldatabase command to see if that helps out my performance. Reference: https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Fri Dec 13, 2019 4:17 pm 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
This JSON grabber is killing me! I can run
Code:
tv_grab_zz_sdjson_sqlite --config-file ~/.mythtv/SD.xmltv --output ~/.mythtv/tv_grab_file.xmltv
and then
Code:
mythfilldatabase --no-allatonce --sourceid 1 --file --xmlfile tv_grab_file.xmltv
and that works. I can run the tv grabber script with a lot of different parameters without seeing an error. However, when I run mythfilldatabase so that it calls the grabber, it exits like this every time:
Code:
2019-12-13 16:40:34.252280 N  Refreshing data for Sun Dec 15 2019
2019-12-13 16:40:34.252564 I  XMLTV config file is: /home/mythtv/.mythtv/SD.xmltv
2019-12-13 16:40:36.591581 E  FillData: XMLTV grabber returned error code 1
2019-12-13 16:40:37.291693 E  Error in 1:1: unexpected end of file
I can't seem to find a log for mythfilldatabase, because as far as I can tell one isn't being generated when it's not being run by the system. I even tried adding --loglevel debug, and mythfilldatabase didn't generate a log.

For now I can manually update the grabber data every so often, but I'd love to find out why it never works automatically. What else can I check?

Also, the channels are driving me crazy. For whatever reason, Comcast broadcasts up to 4 copies of many channels (I've verified on the cable box these exist). I thought I had them all wiped out, but today running the grabber and/or mythfilldatabase returned many dupes and unwanted channels to the list. Going through typing "no" 300 times isn't a lot of fun, but I guess that's the only recourse. I sure wish there was a more user-friendly way to select channels!

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Tue Dec 24, 2019 12:18 am 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
I'm replying here in case the information helps anyone else. As usual, it turns out most of my problems were self induced. TL;DR version: the steps in bold have my mythfilldatabase working for 4 straight days so far.

Following the instructions at https://www.mythtv.org/wiki/XMLTV#MythTV_Setup, I copied the contents of ~/.xmltv and ~/.mythtv/xxx.xmltv to the ~ directory. I didn't think it through at first and now realize a symlink would have been a better idea. When troubleshooting and running commands, sometimes I was in the ~ directory and sometimes not. Apparently it was using one copy of the database when the command was run from the ~ directory, and the other database for other directories. That was causing the inconsistent channel listings I saw. Creating symlinks seems to have fixed this issue.

Additionally, during setup the script offered the default database as ~/.mythtv/SchedulesDirect.DB or something like that. When I named my database "SD.DB' instead, I didn't realize that no directory information was stored in the config file. Every time the system tried to run the grabber, it was looking for the database in <some directory>/SD.DB instead of ~/SD.DB and would fail because it couldn't connect to the database. Changing the xmltv config file to reflect the whole path instead of just the file name seems to have fixed this issue.

Finally, following advice on the mythtv wiki to run mythfilldatabase with "--no-allatonce" seems to have been a bad idea. I'm speculating since the log level isn't specific enough to show me everything that's going on behind the scenes. It seems like the --no-allatonce option causes mythfilldatabase to run the grabber multiple times, each with a different offset day. The problem is the grabber seems to be returning only 18 days of data currently instead of the advertised 21. It seems like when mythfilldatabase tries to ask for data that doesn't exist, it causes an error that stops mythfilldatabase from finishing correctly. Removing the --no-allatonce flag seems to have fixed the issue, though my 4GB RAM is maxed out for about 90 minutes to process 360 channels' worth of data.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Fri Jan 17, 2020 12:35 pm 
Offline
Joined: Mon Jul 01, 2019 5:24 pm
Posts: 10
FYI, my observations after upgrading from DD to the JSON grabber.

Don't be confused when you see these referred to as either JSON grabbers or XMLTV grabbers. I was, because XML and JSON are not the same thing. In summary, the grabber is getting the data from SchedulesDirect in JSON format and then outputting an XMLTV file that MythTV consumes.

You get more days of data with the JSON grabber. Some documentation I've read says 21 days, but I've only seen it get 17 days and so have others I've talked too. So don't think something's wrong if you don't get 21 days of listings.

The data after the upgrade seems consistently better. I had some channels that didn't list shows as HD when they actually were, that are fixed after the upgrade. Also, season and episode numbers were missing for some channels before upgrade and are present after. I don't know why, but it is a plus and a good reason to upgrade.

If you log in as a different user than 'mythtv' (I do on Xubuntu but I don't remember if LinHES allows this) you may have permissions issues running it manually vs. when the system runs it automatically as mythtv.

_________________
Not using LinHES because 8.6 didn't support my hardware, but still very interested in future developments, so lurking once in a while. :)


Top
 Profile  
 
PostPosted: Fri Jan 13, 2023 8:32 am 
Offline
Joined: Sun Sep 10, 2006 2:58 pm
Posts: 2
A big shout out to the maintainers of LinHES. I'm a former Knoppmyth user and am so glad to see this is still carrying on in one form or another.

For me, XMLTV schedules setup was a bit challenging and I ran across this thread and just wanted to clarify things.

Bottom line: mythfilldatabase using the argument "--no-allatonce" works perfectly fine and I use "--refresh 0-21" as well. I think one of the previous poster's problem was with links and permissions.

I am sharing what I did to get things to work and I also posted this in the MythTV Forum. I am "gnurocks" there. See https://forum.mythtv.org/viewtopic.php?f=36&t=5227

What worked for me using Ubuntu MATE 22.04 / Mythtv v32:

I used “Jim A’s” outstanding guide. See http://mythtvinstall.blogspot.com/2018/10/creating-mythtv-v29-frontendbackend-on.html

For my workflow, I ran mythtv-setup as $USER and ran the tv_grab-zz_sdjon_sqlite setup files as mythtv (Jim A’s guide, step 24). Yes, I know of the Mythtv XMLTV Wiki warning about running mythtv-setup as $USER.

For step 26 of Jim A’s guide, “Create links for all the files created by XMLTV setup in /home/mythtv/.xmltv and /home/mythtv/.mythtv in /home/$USER …,” I did the following:

HARD LINK (not symlink) the database file to /home/$USER (allows mythfilldatabase to be run correctly by mythtv or $USER):
Code:
sudo ln /home/mythtv/.xmltv/SchedulesDirect.DB /home/username/.xmltv/SchedulesDirect.DB


Give RW permissions for the mythtv user group to SchedulesDirect.DB (default is read only or 644):
Code:
cd /home/mythtv/.xmltv
sudo chmod 664 SchedulesDirect.DB


COPY (not hard link or symlink) the XXXX.xmltv file to /home/$USER:
Code:
cp /home/mythtv/.mythtv/XXXX.xmltv /home/username/.mythtv/XXXX.xmltv


Edit only the newly copied XXXX.xmltv file that now resides in /home/$USER/.mythtv/:
Code:
nano /home/username/.mythtv/XXXX.xmltv
Edit first line to reflect database=/home/username/.xmltv/SchedulesDirect.DB


Change or verify ownership and permissions of XXXX.xmltv:
Code:
cd /home/username/.mythtv

Code:
sudo chown username:username XXXX.xmltv

Code:
sudo chmod 664 XXXX.xmltv


Follow the rest of Jim A’s checklist and after step 32 (exit back to $USER):

Optimize auto mythfilldatabase downloads by going into mythtv-setup -> General -> Program Schedule Downloading Options -> Automatically Update Program Listings -> Guide Data Arguments, and input the following two arguments: --no-allatonce --refresh 0-21

The argument “--no-allatonce” in conjunction with "-–refresh 0-21,” will acquire tomorrow and future days and will also update all changes to preexisting days, all with minimal memory usage. See #12758 patch post, Peter Bennett’s Comment 7 here: https://code.mythtv.org/trac/ticket/12758

Hope this helps.


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 


All times are UTC - 6 hours




Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu