Author |
Message |
christ
|
Posted: Sun Aug 04, 2013 7:43 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
Not sure if anyone else has seen this before and fixed it. I'm on the latest 7.x load with all updates. All backends and frontends are at the same version. Using mythtv menu equivalents to the below items work properly. If I use mythweb to access things like "Backend Status", "Upcoming Recordings" or "Recorded Programs", it often takes a long time (>1min) and often gives the following php errors: Code: Warning at /data/srv/httpd/htdocs/mythweb/modules/tv/upcoming.php, line 91: !!NoTrans: Invalid argument supplied for foreach()!! Sometimes I get other errors such as Code: User Notice at /usr/share/mythtv/bindings/php/MythBackend.php, line 105: !!NoTrans: Unexpected response to MYTH_PROTO_VERSION '72': !! and sometimes I get the errors and no output for the respective page. This used to work fine until about a few months ago. The only thing I can think that really changed is my storage topology. I moved storage from an overloaded CentOS based box to a brand spank'n new NAS4Free based NAS (BSD Based). Recordings, Video, etc storage is all on the NAS box (it was on the CentOS box before) and accessed via CIFS as before. Although my permissions look correct even for http I tried adding http to various privileged groups but this made no difference. CPU on both the mythtv backend and NAS are close to idle. Likely related to this... when I log into any backend including the master backend, we normally get the tuner status and upcoming listings. Often this fails with an error similar to: Code: E [12341] Python Backend Connection Unexpected answer to ANN Couldn't connect to MythTV service for status It is suspiciously similar. Any thoughts? Any simple way on LinHES to trace php to see where it is hanging? I expect it is sitting on some timers or some transaction is very slow. If I can isolate it I'm sure I can find it.
|
|
Top |
|
|
RacerX
|
Posted: Sun Aug 04, 2013 7:52 pm |
|
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
|
|
Top |
|
|
christ
|
Posted: Sun Aug 04, 2013 8:09 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
Just did an strace on php and I see this after a considerable wait: Code: send(5, "GET /Status/GetStatusHTML HTTP/1"..., 36, MSG_DONTWAIT) = 36 send(5, "Host: aaa.bbb.ccc.ddd:6544\r\n", 25, MSG_DONTWAIT) = 25 send(5, "\r\n", 2, MSG_DONTWAIT) = 2 *** COMMENT: this is where the waiting begins.... poll([{fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout) poll([{fd=5, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 0 (Timeout) poll([{fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout) poll([{fd=5, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=5, revents=POLLIN}]) recv(5, "HTTP/1.0 200 OK\r\nDate: 4 Aug 201"..., 8192, MSG_DONTWAIT) = 8019 close(5) = 0 time(NULL) = 1375668073 time(NULL) = 1375668073 lstat64("/usr/local/share/mythtv/bindings/php//modules/status/tmpl/default/status.php", 0xbfa3f3dc) = -1 ENOENT (No such file or directory)
Looks like it is not getting an answer on socket 6544 when it fails. (note aaa.bbb.ccc.ddd is masking my IP address space) edit: showed in line where the waiting begings.
Last edited by christ on Sun Aug 04, 2013 8:33 pm, edited 1 time in total.
|
|
Top |
|
|
RacerX
|
Posted: Sun Aug 04, 2013 8:15 pm |
|
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
|
Can your run
find / -name MythBackend.php
and share the results
|
|
Top |
|
|
christ
|
Posted: Sun Aug 04, 2013 8:17 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
RacerX wrote: Read thru this it's pretty interesting.... something about the path of MythBackend.php http://www.mythtv.org/pipermail/mythtv- ... 13285.htmlThanks for taking the time. MythBackend.php is in the correct path and there are no duplicates. Specifically: /usr/share/mythtv/bindings/php/MythBackend.phpIn his case his failure was constant. In my case it is not 100% failure. In fact sometimes it works perfectly (but not often). It looks like something is timing out somewhere. Just need to track it down... Hoping someone else has seen it before or has other clues.
|
|
Top |
|
|
christ
|
Posted: Sun Aug 04, 2013 8:21 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
RacerX wrote: Can your run
find / -name MythBackend.php
and share the results See above but for completeness: Code: # find /[a-l]* -name MythBackend\.php # find /[n-z]* -name MythBackend\.php /usr/share/mythtv/bindings/php/MythBackend.php I skip media, mnt, and myth as they should not be relevant and they are large and 2 of which are across the network.
|
|
Top |
|
|
RacerX
|
Posted: Sun Aug 04, 2013 9:07 pm |
|
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
|
The status you get when you access port 6544 does not have anything to do with mythweb can you Code: sudo pkill -9 mythbackend
restart mythbackend and then try to access mythweb
|
|
Top |
|
|
christ
|
Posted: Mon Aug 05, 2013 8:56 am |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
RacerX wrote: The status you get when you access port 6544 does not have anything to do with mythweb Why do you say that? My understanding is that mythweb uses php to send a request to port 6544. The trace above is specifically tracing php when I click "Backend Status". What is your understanding? RacerX wrote: can you Code: sudo pkill -9 mythbackend
restart mythbackend and then try to access mythweb No effect. The first two requests worked but were slow and then the issue reappeared. It is somewhat random in its failure in that it fails perhaps 80% of the time. I redid the pkill/restart and mythweb failed on the first try so I think the fact the first two requests worked on the first restart is likely coincidence.
|
|
Top |
|
|
RacerX
|
Posted: Mon Aug 05, 2013 10:24 am |
|
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
|
What happens when you change your current theme setting?
Can you dig into your backend log and look for errors like Permission denied (13) or Relating to Specific Shows that have problems with Snapshot Creation or do not exist
|
|
Top |
|
|
christ
|
Posted: Mon Aug 05, 2013 1:33 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
RacerX wrote: What happens when you change your current theme setting? No change. RacerX wrote: Can you dig into your backend log and look for errors like Permission denied (13) or Relating to Specific Shows that have problems with Snapshot Creation or do not exist No issues. I've combed through the logs extensively and can find nothing which seems to indicate an issue.
|
|
Top |
|
|
RacerX
|
Posted: Mon Aug 05, 2013 11:51 pm |
|
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
|
|
Top |
|
|
slowtolearn
|
Posted: Tue Aug 06, 2013 1:29 pm |
|
Joined: Wed Nov 16, 2005 8:55 pm
Posts: 1381
Location:
Farmington, MI USA
|
As you mentioned you rearranged your storage you may need to check the permissions on the CIFS mounted directories/files: http://www.gossamer-threads.com/lists/m ... ers/443051I've seen strange issues when the Backend and Master Server IPs don't match in the database: Code: select * from settings where value like "%Server%";
| BackendServerIP | 10.0.0.240 | myth1 | | BackendServerPort | 6543 | myth1 | | MasterServerIP | 10.0.0.240 | NULL | | MasterServerPort | 6543 | NULL | (In the output above myth1 is my Master backend. You may find more BackendServerIP entries, just make sure they all point to your Master backend IP address and that MasterServerIP does as well). However, since the problem does not occur all the time it smells of a name resolution issue. This indicates that at some point IPv6 entries in /etc/hosts could cause problems, and I know I have borked my own setup with incorrect entries in /etc/hosts. Does your /etc/hosts entry for your Master backend match the name from hostname? I have become accustomed to defining both the loopback (127.0.0.1) and physical interfaces, such as Code: 127.0.0.1 somedomain.local ... 10.0.0.240 somedomain.local myth1 christ wrote: Code: send(5, "Host: aaa.bbb.ccc.ddd:6544\r\n", 25, MSG_DONTWAIT) = 25 Looks like it is not getting an answer on socket 6544 when it fails. (note aaa.bbb.ccc.ddd is masking my IP address space) If you run straces on requests that succeed, does the same host show up? In other words, is it always trying to contact the same IP address? Also, what is the contents of your /etc/resolv.conf file? I am not running LinHES currently so I don't know where these may be, but do the httpd logs give any more clues?
|
|
Top |
|
|
christ
|
Posted: Tue Aug 06, 2013 8:00 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
Interesting thoughts slowtolearn. I did see that article wrt permissions. There is nothing obviously wrong with my permissions but for good measure I added http to all of the main groups and that made no difference. I can try brute forcing the permissions on the NAS to be wide open and see if that changes anything. This is a high probability culprit. I checked the db and I have a number of entries due to the history and age of the database. However the master backend's "BackendServerIP" is the same as the NULL "MasterServerIP" as it should be (and as you say). The various slave backends' "BackendServerIP" are their own IP addresses which I believe is correct. My /etc/hosts is simple and has no IPv6 info in it. It is simply: Code: 127.0.0.1 mythtvmbe localhost I believe this is correct though I could try changing it to the absolute IP address and check. The address aaa.bbb.ccc.ddd is the correct IP address for the master backend/mysql db server. The IP address is the same for the success path. As for resolv.conf, it looks correct... Code: # cat /etc/resolv.conf # Generated by dhcpcd from eth0 # /etc/resolv.conf.head can replace this line domain <mydomain.com> nameserver <ip of domain server> # /etc/resolv.conf.tail can replace this line I have a separate server that resolves all local hostnames. This server has the correct entry for mythvmbe (mythtvmbe.mydomain.com ->aaa.bbb.ccc.ddd) I wish there was an easy way to trace what is happening within mythbackend when the command goes to port 6543. But the backend is always really busy doing stuff which generates tonnes of noise! Thanks for the fresh ideas! It gives me other avenues to pursue. My inclination at this stage is to move the master backend to R8 and see if the problem remains. The nature by which R8 upgrades gives it a pretty fresh environment getting rid of a number of variables.
|
|
Top |
|
|
christ
|
Posted: Tue Aug 06, 2013 8:20 pm |
|
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location:
Ottawa, Canada
|
RacerX wrote: There something strange about your setup I suspect these are artifacts of the timeout where the code is borking on default values. I think one of next steps will be to shut down everything that is not essential to mythtv on my network. My network is more complicated than the average home user! It includes: A router with name server (CentOS based) A really big NAS (NAS4Free) An ESXi 5.1 server which hosts the following servers (iSCSI to the NAS for the VMs): - Web/Storage server (CentOS based) - Master Backend - two Slave backends used for commercial skip - Zoneminder - some test VMs A physical Slave backend with 2 HD-PVRs An IP Telephony server (commercial) A Session Border Controller (SBC) for my SIP connections (commercial) 4 Mythtv frontends about 12 VoIP phones several IP cameras About 10 PCs/MACs/Tablets/Smartphones Appliances such as printers, TVs, game consoles, Surround Processors, Receivers, etc. And of course a GigE layer 2 network to pull it together. I probably forgot some stuff too.
|
|
Top |
|
|
slowtolearn
|
Posted: Wed Aug 07, 2013 6:59 am |
|
Joined: Wed Nov 16, 2005 8:55 pm
Posts: 1381
Location:
Farmington, MI USA
|
christ wrote: My /etc/hosts is simple and has no IPv6 info in it. It is simply: Code: 127.0.0.1 mythtvmbe localhost I believe this is correct though I could try changing it to the absolute IP address and check. Worth a shot, although I would add an additional line like Code: aaa.bbb.ccc.ddd mythtvmbe OR aaa.bbb.ccc.ddd <domain> mythtvmbe christ wrote: As for resolv.conf, it looks correct... Code: # cat /etc/resolv.conf # Generated by dhcpcd from eth0 # /etc/resolv.conf.head can replace this line domain <mydomain.com> nameserver <ip of domain server> # /etc/resolv.conf.tail can replace this line I have a separate server that resolves all local hostnames. This server has the correct entry for mythvmbe (mythtvmbe.mydomain.com ->aaa.bbb.ccc.ddd) Was/is the master backend using DHCP to acquire an IP address? I ask because of the comments in your resolv.conf file. Or, is it possible your master backend's IP address is within your DHCP range, and another device is attempting to use that same address? christ wrote: I wish there was an easy way to trace what is happening within mythbackend when the command goes to port 6543. But the backend is always really busy doing stuff which generates tonnes of noise! If all of your IP traffic goes through your CentOS router it would be simple to track from there (or from the master backend itself, just use the "port" option): Code: tcpdump -ni ethX host aaa.bbb.ccc.ddd and port 6543 May need to add/play with some options, such as -W to create a file for use with Wireshark to help in diagnosis, maybe the -v, -vv or -vvv options. christ wrote: Thanks for the fresh ideas! It gives me other avenues to pursue. No problem, hope you get it sorted out.
|
|
Top |
|
|