LinHES Forums

JSON grabber instad of the Data Direct grabber?
Page 1 of 1

Author:  mattbatt [ Mon Jun 10, 2019 6:00 pm ]
Post subject:  JSON grabber instad of the Data Direct grabber?

I saw in another thread that brfransen was saying we will be changing thew way we get our lineup.
brfransen wrote:
If you are using the "North American ( (Internal)" grabber press the right arrow and enter the user id an password there. But that grabber is going to be removed for mythtv v31 so it is recommended to use one of the xmltv SD JSON grabbers. See ... rect_Setup

So doing some digging in the setup page I can't figure out how to configure this from the GUI. Is it possible or is the only option in the command line?
If someone more knowledgeable about this than I am could make a step by step guide (preferably with lots of colorful pictures) on how all of us Schedules Direct users will need to get our lineup in the near(ish) future that would be awesome.


Author:  brfransen [ Wed Jun 12, 2019 6:42 am ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

The MythTV Wiki I linked is the most complete guide I have seen. I think all of the XMLTV grabbers are configured via CLI.

Author:  mattbatt [ Fri Jun 14, 2019 10:33 pm ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

Yikes! That is less user friendly than Data Direct.

Author:  welner [ Tue Aug 13, 2019 3:43 pm ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

As I live in the north east, I was patiently waiting for phase 2 of the repack to complete before attempting this change.

This gave me some time to read and reread the instructions.

the instructions are accurate, but only become clearer once you perform the steps.

the one thing that slowed me down was a few mystery nonexistent channels that appeared after switching grabbers, and were not added during the channel rescan i did for the repack a couple weeks ago (they also did not exist before the repack). these channels were not listed in SchedulesDirect. using the channel editor in mythweb, i would delete them. unfortunately, those channels reappeared after running mythfilldatabase.

this part of the instructions helped me get rid of those bogus channels:

Run these:

tv_grab_zz_sdjson_sqlite --days 0 --config-file $HOME/.mythtv/SD.xmltv
tv_grab_zz_sdjson_sqlite --manage-lineups --config-file $HOME/.mythtv/SD.xmltv
If you get all channels in the lineup, then select the options that will make current and future channels selected.
If you only get a few of the channels you will need to select the appropriate channels. Select a default for existing and future channels, then use option 5 to select which channels you need. This goes though all channels one at a time. You have to complete this for every channel in the list. You cannot stop partway through.
If you need to do the channel selection again because of errors or channel changes, your previous values are remembered and used as defaults.

i performed the second "if" clause in the instructions and was able to delete the bogus channels. in my case, i had too many channels, not too few. so, it was not intuitive that i needed to perform option 5. after rerunning mythfilldatabase, the channels did not reappear.

in the nit-picking department, one of the new features of this grabber is that you get channel icons displayed in the listing on mythweb and in the program guide in myth. unfortunately, some of the icons are black, and as a result are not visible on the black or dark blue backgrounds. here is one example icon that is not visible:

Author:  mattbatt [ Fri Aug 30, 2019 11:15 pm ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

OK the DataDirect mode stopped working recently and apparently ran out of data TODAY!!!
I'm not sure what broke but it no longer liked my username and password.
SO I was forced to update to the new JSON mode and boy did I suck at it 2 hours later and it's finally working. Try this first then move on to command line.
1) I hate the new setup GUI. They made it so hard to enter answers.
2) the order of the guide is way out of wack, at least I think. I ended up doing most of the steps 3 times because it didn't work.
3) when you go into Video Source setup and you select the source you want to edit (or create) the "Listings Grabber" you want is at the end of the list called "Schedules Direct JSON API (xmltv)"
4) it seems like using the GUI in Mythtv-Setup is really just going through the same command line steps except that it works (when I finally selected the correct grabber)
5) the guide says that if you use ATSC tuner you will have to manually edit the channel IDs. When you do it though the Mythtv-Setup GUI it worked fine.
6) I might have to spin up another VM to do a step by step walk through.

Author:  DoomedTX [ Thu Dec 12, 2019 8:38 am ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

Can anyone help troubleshoot this? I followed the instructions to setup the sqlite3 grabber, and all seemed to go well. All the channels, channel icons, and 21 days of data were downloaded. However, every day since then mythfilldatabase fails with the same error. I ran it last night, and here is the relevant snippet of the log from that run:

2019-12-11T13:01:35.536901-05:00 mythfilldatabase[7028]: I CoreContext filldata.cpp:379 (GrabData) XMLTV config file is: /home/mythtv/.mythtv/SD.xmltv
2019-12-11T13:01:37.539270-05:00 mythfilldatabase[7028]: E CoreContext filldata.cpp:441 (GrabData) FillData: XMLTV grabber returned error code 255
2019-12-11T13:01:38.039736-05:00 mythfilldatabase[7028]: E CoreContext xmltvparser.cpp:672 (parseFile) Error in 1:1: unexpected end of file

Clearly the grabber doesn't like something, but I can't find anything on the internet about code 255. I re-ran the grabber setup, and it looked good with the correct lineup and password. Running mythfilldatabase -v in a terminal window didn't show anything helpful. Is there anything else I can check to start getting valid guide data again?

Edit: running the grabber manually now gets this error, though it worked the first time with days=0:

tv_grab_zz_sdjson_sqlite --days 21 --config-file $HOME/.mythtv/SD.xmltv
WARNING: Perl module JSON::XS not installed.  JSON encode/decode performance will be poor.
Opening the local database
Can't call method "errstr" on an undefined value at /usr/bin/vendor_perl/tv_grab_zz_sdjson_sqlite line 3857

Author:  welner [ Thu Dec 12, 2019 8:52 am ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

i would suggest looking at the file


it seems as though the "parse error" message relates to that file

Author:  DoomedTX [ Thu Dec 12, 2019 10:17 am ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

Thanks. It seems like the error is in the grabber script, specifically line 3857. Here's the surrounding code there:

$DBH = DBI->connect("DBI:SQLite:dbname=$dbname", "", "",
                        { RaiseError => 0, PrintError => 0, AutoCommit => 0 });

    if (!defined($DBH))
        print (STDERR "Unable to open database file $dbname: " . $DBH->errstr . "\n");

From what I've been able to piece together, it seems the database is failing to connect to the database in 3852. That is leaving $DBH undefined rather than false. When it gets to 3857 and attempts "$DBH->errstr," it gives the error "Can't call method "errstr" on an undefined value." So, as far as I can tell, the underlying problem is that it isn't connecting to the database. I'm not sure why it connected to the database just fine the first time I ran the setup, but I'll poke through the setup again to see if I can figure out where it's going wrong.

I'll throw out another tidbit I just learned since I've noticed my system practically shuts down during mythfilldatabase:
When using XMLTV, mythfilldatabase can use a lot of main memory. If you have 200 or more channels and you are processing 21 days of schedules it can use 5 GB of resident memory.
I'm going to be adding the --no-allatonce option to my mythfilldatabase command to see if that helps out my performance. Reference:

Author:  DoomedTX [ Fri Dec 13, 2019 4:17 pm ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

This JSON grabber is killing me! I can run
tv_grab_zz_sdjson_sqlite --config-file ~/.mythtv/SD.xmltv --output ~/.mythtv/tv_grab_file.xmltv
and then
mythfilldatabase --no-allatonce --sourceid 1 --file --xmlfile tv_grab_file.xmltv
and that works. I can run the tv grabber script with a lot of different parameters without seeing an error. However, when I run mythfilldatabase so that it calls the grabber, it exits like this every time:
2019-12-13 16:40:34.252280 N  Refreshing data for Sun Dec 15 2019
2019-12-13 16:40:34.252564 I  XMLTV config file is: /home/mythtv/.mythtv/SD.xmltv
2019-12-13 16:40:36.591581 E  FillData: XMLTV grabber returned error code 1
2019-12-13 16:40:37.291693 E  Error in 1:1: unexpected end of file
I can't seem to find a log for mythfilldatabase, because as far as I can tell one isn't being generated when it's not being run by the system. I even tried adding --loglevel debug, and mythfilldatabase didn't generate a log.

For now I can manually update the grabber data every so often, but I'd love to find out why it never works automatically. What else can I check?

Also, the channels are driving me crazy. For whatever reason, Comcast broadcasts up to 4 copies of many channels (I've verified on the cable box these exist). I thought I had them all wiped out, but today running the grabber and/or mythfilldatabase returned many dupes and unwanted channels to the list. Going through typing "no" 300 times isn't a lot of fun, but I guess that's the only recourse. I sure wish there was a more user-friendly way to select channels!

Author:  DoomedTX [ Tue Dec 24, 2019 12:18 am ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

I'm replying here in case the information helps anyone else. As usual, it turns out most of my problems were self induced. TL;DR version: the steps in bold have my mythfilldatabase working for 4 straight days so far.

Following the instructions at, I copied the contents of ~/.xmltv and ~/.mythtv/xxx.xmltv to the ~ directory. I didn't think it through at first and now realize a symlink would have been a better idea. When troubleshooting and running commands, sometimes I was in the ~ directory and sometimes not. Apparently it was using one copy of the database when the command was run from the ~ directory, and the other database for other directories. That was causing the inconsistent channel listings I saw. Creating symlinks seems to have fixed this issue.

Additionally, during setup the script offered the default database as ~/.mythtv/SchedulesDirect.DB or something like that. When I named my database "SD.DB' instead, I didn't realize that no directory information was stored in the config file. Every time the system tried to run the grabber, it was looking for the database in <some directory>/SD.DB instead of ~/SD.DB and would fail because it couldn't connect to the database. Changing the xmltv config file to reflect the whole path instead of just the file name seems to have fixed this issue.

Finally, following advice on the mythtv wiki to run mythfilldatabase with "--no-allatonce" seems to have been a bad idea. I'm speculating since the log level isn't specific enough to show me everything that's going on behind the scenes. It seems like the --no-allatonce option causes mythfilldatabase to run the grabber multiple times, each with a different offset day. The problem is the grabber seems to be returning only 18 days of data currently instead of the advertised 21. It seems like when mythfilldatabase tries to ask for data that doesn't exist, it causes an error that stops mythfilldatabase from finishing correctly. Removing the --no-allatonce flag seems to have fixed the issue, though my 4GB RAM is maxed out for about 90 minutes to process 360 channels' worth of data.

Author:  wesmorrison [ Fri Jan 17, 2020 12:35 pm ]
Post subject:  Re: JSON grabber instad of the Data Direct grabber?

FYI, my observations after upgrading from DD to the JSON grabber.

Don't be confused when you see these referred to as either JSON grabbers or XMLTV grabbers. I was, because XML and JSON are not the same thing. In summary, the grabber is getting the data from SchedulesDirect in JSON format and then outputting an XMLTV file that MythTV consumes.

You get more days of data with the JSON grabber. Some documentation I've read says 21 days, but I've only seen it get 17 days and so have others I've talked too. So don't think something's wrong if you don't get 21 days of listings.

The data after the upgrade seems consistently better. I had some channels that didn't list shows as HD when they actually were, that are fixed after the upgrade. Also, season and episode numbers were missing for some channels before upgrade and are present after. I don't know why, but it is a plus and a good reason to upgrade.

If you log in as a different user than 'mythtv' (I do on Xubuntu but I don't remember if LinHES allows this) you may have permissions issues running it manually vs. when the system runs it automatically as mythtv.

Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group