Frequent reflector connection errors

Posted on
Fri Apr 03, 2020 8:35 pm
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

Great idea, Matt! I can also test out my theory that https might work where http didn't work to keep the persistent connection alive. I'll try various things and let you know what happens. In any event, this should definitely provide a solution. Thanks!

--Dave

Posted on
Fri Apr 03, 2020 8:44 pm
matt (support) offline
Site Admin
User avatar
Posts: 19964
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

Definitely test the theory, but I believe it is going to be the duration between running the script and not HTTP vs HTTPS that makes the difference. By the time the command is sent back to your Indigo Web Server it is over an encrypted tunnel regardless of if HTTP or HTTPS was used. Let me know how it works out.

Image

Posted on
Sat Apr 04, 2020 10:49 pm
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

I'm still running tests and am getting some inconsistent results. Initially, running the test with an embedded script was timing out sometimes so I converted to an external script which seems to work. Since it still triggers communication back through the tunnel, this should give the desired result too, right?

I'll continue trying to find timing that works consistently. I'll also try running the server through my iPhone hotspot which takes my router and ISP out of the picture. This all takes lots of time since I have to wait 2.5 hours with every new trial.

Posted on
Sun Apr 05, 2020 12:23 pm
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

I got no connection errors after 6 hours using cell connection with a personal hotspot and no repeating http messages. Looks like xfinity (or possibly my router) is indeed terminating the socket after 2.5 hours.

I'll keep searching for a reliable way to prevent this. Going down as low as 10 seconds on the https message interval didn't work. It seems to have made it worse actually, causing repeated failures in reestablishing the connection. This might have something to do with the timeouts I experienced when using an embedded script for sending the messages.

Things get a little complicated on the tests since I have no way to turn off the 10 minute checks the server performs automatically.

Posted on
Mon Apr 06, 2020 1:28 pm
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

It's taking a while to find an optimal interval for the repeating messages. Too long and they don't help. Too short and they seem to make things worse by triggering more errors and causing the reflector reconnections to fail. 2 minutes seems like it might be a good repetition interval for https messages. I haven't had an error for 6 hours whereas I would have had 2 errors in that time without the script running.

I'll keep an eye on it but this is likely to be acceptable even if I get occasional errors. Of course, I still don't know what is causing the error but it must have something to do with my router, modem or Comcast ISP.

Since all 3 of these devices are very common, I'm surprised others are not reporting the problem. If this does seem to be a problem with others, it would be great if you could reduce the check interval to something like 2 minutes or even make it configurable to accommodate possibly different timeout values among ISPs. This would have the added benefit of detecting and correcting the issue earlier when a failure does happen.

Thanks for your help.

--Dave

Posted on
Mon Apr 06, 2020 2:01 pm
matt (support) offline
Site Admin
User avatar
Posts: 19964
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

It is odd that having too short of an interval (< 2 minutes?) causes errors with the reflector connection failing. What errors are shown in the Event Log (both from the script you are running and Indigo Server trying to recreate the reflector connection)? I say it is odd because leaving a Web browser up will cause traffic ever few seconds and it didn't sound like you had problems in that scenario. I don't know what would be different from the browser making an HTTP/S call (via JavaScript) versus your Python script making an HTTP/S call.

This really isn't a problem we've had with anyone else that hasn't been resolved except the discussion on this thread, so it doesn't really move the needle on the frequent-problems list. I think maybe we've had 3 or 4 other cases of reflector connectivity issues in the past, but those were solved with changing router settings, replacing the router, or in one case an Indigo change we made because a satellite ISP would totally block the reflector connection request (it would never work because they blocked the port).

Image

Posted on
Mon Apr 06, 2020 2:37 pm
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

matt (support) wrote:
It is odd that having too short of an interval (< 2 minutes?) causes errors with the reflector connection failing. What errors are shown in the Event Log (both from the script you are running and Indigo Server trying to recreate the reflector connection)? I say it is odd because leaving a Web browser up will cause traffic ever few seconds and it didn't sound like you had problems in that scenario. I don't know what would be different from the browser making an HTTP/S call (via JavaScript) versus your Python script making an HTTP/S call.

When the timing gets down in the range of 5-10 seconds, it takes multiple reconnection attempts, sometimes reaching the 15 minute level. The server logs the error in the usual way, showing the connection test failed and the reconnection scheduled after a specified time. The script shows "Reflector connection is currently down" at whatever interval it's using while the reflector is down. I agree it's odd that the script would do this but the browser would not. I can confidently report it but I can't explain it. Some of the inconsistencies may have to do with the relative timing of the reflector checks and the script executions. Also, the browser is sending a few device status requests whereas the script is sending a prismstatus message. I'll try changing the url in the script and browser to see if that changes anything. (Edit: Whoops, I couldn't make them equivalent since the prismstatus url doesn't repeat in the browser and the control page url won't authenticate in the script.)

matt (support) wrote:
This really isn't a problem we've had with anyone else that hasn't been resolved except the discussion on this thread, so it doesn't really move the needle on the frequent-problems list. I think maybe we've had 3 or 4 other cases of reflector connectivity issues in the past, but those were solved with changing router settings, replacing the router, or in one case an Indigo change we made because a satellite ISP would totally block the reflector connection request (it would never work because they blocked the port).

I understand completely you not changing the software in this circumstance. Is it possible that the 10 minute reflector check interval is exposed in a parameter list somewhere that I could modify as a test?

Posted on
Mon Apr 06, 2020 3:37 pm
matt (support) offline
Site Admin
User avatar
Posts: 19964
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

DVDDave wrote:
Is it possible that the 10 minute reflector check interval is exposed in a parameter list somewhere that I could modify as a test?

Unfortunately, it is hard coded.

Image

Posted on
Wed Apr 08, 2020 10:55 am
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

One more FYI: I haven't gotten a reflector error for the last 24 hours which is far longer than I've seen since the problem began. I'm running the prismstatus script every 2 minutes.

I can only think of one thing I did differently. I was experimenting with changing the default http timeout value to as high as 630 seconds which was the last value I tried. It didn't seem to have an effect so I restarted the server and went back to a 2 minute interval of https messages without the timeout command. I mention this since I'm unclear on how the default timeout works and how persistent it may be. I doubt it is having an effect but I mention it just in case.

In any event, I'll stop experimenting now since I don't want to risk messing things up again. Hopefully this solution will survive any updates/restarts I need to make in the future.

--Dave

Posted on
Thu Apr 09, 2020 4:01 pm
DVDDave offline
Posts: 373
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

Another big surprise: I stopped the prismstatus script from running and am still getting no errors! This makes it much more likely that changing the default timeout to a little more than 10 minutes and then restarting the reflector connection is indeed having an effect. Maybe Xfinity handles connection timeouts differently than you're used to. Or this is a huge coincidence after getting the error so consistently and for so long. :wink:

I'll see if the fix persists after any reboots, changes in addresses, etc as they happen, and then add the code to my server startup actions if necessary. Code looks like this:
Code: Select all
# timeout in seconds
timeout = 630
socket.setdefaulttimeout(timeout)

Who is online

Users browsing this forum: No registered users and 1 guest