View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 16 posts ] 
Go to page 1, 2  Next

Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Sun Aug 04, 2013 7:43 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
Not sure if anyone else has seen this before and fixed it.

I'm on the latest 7.x load with all updates. All backends and frontends are at the same version.

Using mythtv menu equivalents to the below items work properly.

If I use mythweb to access things like "Backend Status", "Upcoming Recordings" or "Recorded Programs", it often takes a long time (>1min) and often gives the following php errors:
Code:
Warning at /data/srv/httpd/htdocs/mythweb/modules/tv/upcoming.php, line 91:
!!NoTrans: Invalid argument supplied for foreach()!!

Sometimes I get other errors such as
Code:
User Notice at /usr/share/mythtv/bindings/php/MythBackend.php, line 105:
!!NoTrans: Unexpected response to MYTH_PROTO_VERSION '72': !!

and sometimes I get the errors and no output for the respective page.

This used to work fine until about a few months ago. The only thing I can think that really changed is my storage topology. I moved storage from an overloaded CentOS based box to a brand spank'n new NAS4Free based NAS (BSD Based). Recordings, Video, etc storage is all on the NAS box (it was on the CentOS box before) and accessed via CIFS as before.

Although my permissions look correct even for http I tried adding http to various privileged groups but this made no difference.

CPU on both the mythtv backend and NAS are close to idle.

Likely related to this... when I log into any backend including the master backend, we normally get the tuner status and upcoming listings. Often this fails with an error similar to:
Code:
E [12341] Python Backend Connection Unexpected answer to ANN
Couldn't connect to MythTV service for status

It is suspiciously similar.

Any thoughts?

Any simple way on LinHES to trace php to see where it is hanging? I expect it is sitting on some timers or some transaction is very slow. If I can isolate it I'm sure I can find it.


Top
 Profile  
 
PostPosted: Sun Aug 04, 2013 7:52 pm 
Offline
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
Read thru this it's pretty interesting.... something about the path of MythBackend.php

http://www.mythtv.org/pipermail/mythtv- ... 13285.html


Top
 Profile  
 
PostPosted: Sun Aug 04, 2013 8:09 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
Just did an strace on php and I see this after a considerable wait:
Code:
send(5, "GET /Status/GetStatusHTML HTTP/1"..., 36, MSG_DONTWAIT) = 36
send(5, "Host: aaa.bbb.ccc.ddd:6544\r\n", 25, MSG_DONTWAIT) = 25
send(5, "\r\n", 2, MSG_DONTWAIT)        = 2
*** COMMENT: this is where the waiting begins....
poll([{fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=5, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 0 (Timeout)
poll([{fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=5, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=5, revents=POLLIN}])
recv(5, "HTTP/1.0 200 OK\r\nDate: 4 Aug 201"..., 8192, MSG_DONTWAIT) = 8019
close(5)                                = 0
time(NULL)                              = 1375668073
time(NULL)                              = 1375668073
lstat64("/usr/local/share/mythtv/bindings/php//modules/status/tmpl/default/status.php", 0xbfa3f3dc) = -1 ENOENT (No such file or directory)

Looks like it is not getting an answer on socket 6544 when it fails. (note aaa.bbb.ccc.ddd is masking my IP address space)

edit: showed in line where the waiting begings.


Last edited by christ on Sun Aug 04, 2013 8:33 pm, edited 1 time in total.


Top
 Profile  
 
PostPosted: Sun Aug 04, 2013 8:15 pm 
Offline
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
Can your run

find / -name MythBackend.php

and share the results


Top
 Profile  
 
PostPosted: Sun Aug 04, 2013 8:17 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
RacerX wrote:
Read thru this it's pretty interesting.... something about the path of MythBackend.php

http://www.mythtv.org/pipermail/mythtv- ... 13285.html

Thanks for taking the time. MythBackend.php is in the correct path and there are no duplicates.
Specifically: /usr/share/mythtv/bindings/php/MythBackend.php

In his case his failure was constant. In my case it is not 100% failure. In fact sometimes it works perfectly (but not often). It looks like something is timing out somewhere. Just need to track it down...

Hoping someone else has seen it before or has other clues.


Top
 Profile  
 
PostPosted: Sun Aug 04, 2013 8:21 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
RacerX wrote:
Can your run

find / -name MythBackend.php

and share the results

See above but for completeness:
Code:
# find /[a-l]* -name  MythBackend\.php
# find /[n-z]* -name  MythBackend\.php
/usr/share/mythtv/bindings/php/MythBackend.php

I skip media, mnt, and myth as they should not be relevant and they are large and 2 of which are across the network.


Top
 Profile  
 
PostPosted: Sun Aug 04, 2013 9:07 pm 
Offline
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
The status you get when you access port 6544 does not have anything to do with mythweb

can you

Code:
sudo pkill -9 mythbackend

restart mythbackend and then try to access mythweb


Top
 Profile  
 
PostPosted: Mon Aug 05, 2013 8:56 am 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
RacerX wrote:
The status you get when you access port 6544 does not have anything to do with mythweb

Why do you say that? My understanding is that mythweb uses php to send a request to port 6544. The trace above is specifically tracing php when I click "Backend Status". What is your understanding?

RacerX wrote:
can you

Code:
sudo pkill -9 mythbackend

restart mythbackend and then try to access mythweb

No effect. The first two requests worked but were slow and then the issue reappeared. It is somewhat random in its failure in that it fails perhaps 80% of the time. I redid the pkill/restart and mythweb failed on the first try so I think the fact the first two requests worked on the first restart is likely coincidence.


Top
 Profile  
 
PostPosted: Mon Aug 05, 2013 10:24 am 
Offline
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
What happens when you change your current theme setting?

Can you dig into your backend log and look for errors like
Permission denied (13) or Relating to Specific Shows that have problems with Snapshot Creation or do not exist


Top
 Profile  
 
PostPosted: Mon Aug 05, 2013 1:33 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
RacerX wrote:
What happens when you change your current theme setting?

No change.

RacerX wrote:
Can you dig into your backend log and look for errors like
Permission denied (13) or Relating to Specific Shows that have problems with Snapshot Creation or do not exist

No issues.

I've combed through the logs extensively and can find nothing which seems to indicate an issue.


Top
 Profile  
 
PostPosted: Mon Aug 05, 2013 11:51 pm 
Offline
Joined: Sun Sep 05, 2004 7:06 pm
Posts: 690
There something strange about your setup, what's with

ANN?

http://www.mythtv.org/wiki/ANN_%28Myth_Protocol%29

and MYTH_PROTO_VERSION?

http://www.mythtv.org/wiki/MYTH_PROTO_V ... rotocol%29


Top
 Profile  
 
PostPosted: Tue Aug 06, 2013 1:29 pm 
Offline
Joined: Wed Nov 16, 2005 8:55 pm
Posts: 1381
Location: Farmington, MI USA
As you mentioned you rearranged your storage you may need to check the permissions on the CIFS mounted directories/files:
http://www.gossamer-threads.com/lists/m ... ers/443051

I've seen strange issues when the Backend and Master Server IPs don't match in the database:

Code:
select * from settings where value like "%Server%";

| BackendServerIP                                    | 10.0.0.240                           | myth1     |
| BackendServerPort                                  | 6543                                 | myth1     |
| MasterServerIP                                     | 10.0.0.240                           | NULL      |
| MasterServerPort                                   | 6543                                 | NULL      |
(In the output above myth1 is my Master backend. You may find more BackendServerIP entries, just make sure they all point to your Master backend IP address and that MasterServerIP does as well).

However, since the problem does not occur all the time it smells of a name resolution issue. This indicates that at some point IPv6 entries in /etc/hosts could cause problems, and I know I have borked my own setup with incorrect entries in /etc/hosts. Does your /etc/hosts entry for your Master backend match the name from hostname? I have become accustomed to defining both the loopback (127.0.0.1) and physical interfaces, such as
Code:
127.0.0.1       somedomain.local
...
10.0.0.240      somedomain.local   myth1


christ wrote:
Code:
send(5, "Host: aaa.bbb.ccc.ddd:6544\r\n", 25, MSG_DONTWAIT) = 25

Looks like it is not getting an answer on socket 6544 when it fails. (note aaa.bbb.ccc.ddd is masking my IP address space)

If you run straces on requests that succeed, does the same host show up? In other words, is it always trying to contact the same IP address? Also, what is the contents of your /etc/resolv.conf file?

I am not running LinHES currently so I don't know where these may be, but do the httpd logs give any more clues?


Top
 Profile  
 
PostPosted: Tue Aug 06, 2013 8:00 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
Interesting thoughts slowtolearn.

I did see that article wrt permissions. There is nothing obviously wrong with my permissions but for good measure I added http to all of the main groups and that made no difference. I can try brute forcing the permissions on the NAS to be wide open and see if that changes anything. This is a high probability culprit.

I checked the db and I have a number of entries due to the history and age of the database. However the master backend's "BackendServerIP" is the same as the NULL "MasterServerIP" as it should be (and as you say). The various slave backends' "BackendServerIP" are their own IP addresses which I believe is correct.

My /etc/hosts is simple and has no IPv6 info in it. It is simply:
Code:
127.0.0.1 mythtvmbe localhost

I believe this is correct though I could try changing it to the absolute IP address and check.

The address aaa.bbb.ccc.ddd is the correct IP address for the master backend/mysql db server. The IP address is the same for the success path.

As for resolv.conf, it looks correct...
Code:
# cat /etc/resolv.conf
# Generated by dhcpcd from eth0
# /etc/resolv.conf.head can replace this line
domain <mydomain.com>
nameserver <ip of domain server>
# /etc/resolv.conf.tail can replace this line

I have a separate server that resolves all local hostnames. This server has the correct entry for mythvmbe (mythtvmbe.mydomain.com ->aaa.bbb.ccc.ddd)

I wish there was an easy way to trace what is happening within mythbackend when the command goes to port 6543. But the backend is always really busy doing stuff which generates tonnes of noise!

Thanks for the fresh ideas! It gives me other avenues to pursue.

My inclination at this stage is to move the master backend to R8 and see if the problem remains. The nature by which R8 upgrades gives it a pretty fresh environment getting rid of a number of variables.


Top
 Profile  
 
PostPosted: Tue Aug 06, 2013 8:20 pm 
Offline
Joined: Mon Dec 24, 2007 9:47 am
Posts: 535
Location: Ottawa, Canada
RacerX wrote:
There something strange about your setup

I suspect these are artifacts of the timeout where the code is borking on default values.

I think one of next steps will be to shut down everything that is not essential to mythtv on my network.

My network is more complicated than the average home user! :D

It includes:
A router with name server (CentOS based)
A really big NAS (NAS4Free)
An ESXi 5.1 server which hosts the following servers (iSCSI to the NAS for the VMs):
- Web/Storage server (CentOS based)
- Master Backend
- two Slave backends used for commercial skip
- Zoneminder
- some test VMs
A physical Slave backend with 2 HD-PVRs
An IP Telephony server (commercial)
A Session Border Controller (SBC) for my SIP connections (commercial)
4 Mythtv frontends
about 12 VoIP phones
several IP cameras
About 10 PCs/MACs/Tablets/Smartphones
Appliances such as printers, TVs, game consoles, Surround Processors, Receivers, etc.

And of course a GigE layer 2 network to pull it together. I probably forgot some stuff too.


Top
 Profile  
 
PostPosted: Wed Aug 07, 2013 6:59 am 
Offline
Joined: Wed Nov 16, 2005 8:55 pm
Posts: 1381
Location: Farmington, MI USA
christ wrote:
My /etc/hosts is simple and has no IPv6 info in it. It is simply:
Code:
127.0.0.1 mythtvmbe localhost

I believe this is correct though I could try changing it to the absolute IP address and check.
Worth a shot, although I would add an additional line like
Code:
aaa.bbb.ccc.ddd   mythtvmbe
OR
aaa.bbb.ccc.ddd   <domain>   mythtvmbe


christ wrote:
As for resolv.conf, it looks correct...
Code:
# cat /etc/resolv.conf
# Generated by dhcpcd from eth0
# /etc/resolv.conf.head can replace this line
domain <mydomain.com>
nameserver <ip of domain server>
# /etc/resolv.conf.tail can replace this line

I have a separate server that resolves all local hostnames. This server has the correct entry for mythvmbe (mythtvmbe.mydomain.com ->aaa.bbb.ccc.ddd)
Was/is the master backend using DHCP to acquire an IP address? I ask because of the comments in your resolv.conf file. Or, is it possible your master backend's IP address is within your DHCP range, and another device is attempting to use that same address?

christ wrote:
I wish there was an easy way to trace what is happening within mythbackend when the command goes to port 6543. But the backend is always really busy doing stuff which generates tonnes of noise!
If all of your IP traffic goes through your CentOS router it would be simple to track from there (or from the master backend itself, just use the "port" option):
Code:
tcpdump -ni ethX host aaa.bbb.ccc.ddd and port 6543
May need to add/play with some options, such as -W to create a file for use with Wireshark to help in diagnosis, maybe the -v, -vv or -vvv options.

christ wrote:
Thanks for the fresh ideas! It gives me other avenues to pursue.
No problem, hope you get it sorted out.


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 16 posts ] 
Go to page 1, 2  Next



All times are UTC - 6 hours




Who is online

Users browsing this forum: No registered users and 91 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu