Frequent reflector connection errors

Understood and I will certainly work on that. My son who lives across the country just told me he gets similar ping timeouts with google but is getting none with indigodomo.

That said, what could cause a random error like packet loss to turn into a very consistent error that happens like clockwork every 2.5 hours? I really think we're dealing with 2 different issues and will continue doing the tests you requested.

Thanks for your patience!

--Dave

Open a Terminal window and start a ping to the reflector server:

Code: Select all

ping indigodomo.net

Leave that up and the next time there is a reflector failure see if there were ping failures around that time too.

The reflector connection error happened right on schedule while pinging. There were no ping timeouts at that time. Of course, the timeouts I mentioned before were continuing but not at that exact time.

Next step: Open the browser window overnight to see if I get any connection errors.

Checking back in on this, right now, I'm getting occasional disconnects at random times. Only one in the last 24 hours. When I had the browser window open, I also got random errors but more frequently.

That said, I have no idea what changed to stop the very consistent disconnects. However, since they don't happen often, I'm going to leave it alone for now.

Thanks for your help.

--Dave

A quick and encouraging update on my situation:

In the past few days I made two changes that might have fixed the problem for me—unless there is some other coincidental change I don't know about... I did not carry out a controlled application of each change to try and identify the cause—I can't do that, I just needed for it to work so applied several changes at once.

1. I added a new cable modem/router (Netgear C3700v2) in front of my other router. The new router supports NAT Loop Back, whereas my existing one (which is now behind the new one) does not.

2. I upgraded my home Internet (like everyone else these days—it's suddenly more important) which includes greater bandwidth. I did not measure latencies, etc. and I'm guessing this has little/nothing to do with what I was seeing, but I did change it.

Ever since I got the routers in place and configured (DHCP reservations, port forwarding through two routers, etc.), I have not seen one failure. The reflector via Indigo Touch, web, etc. has worked flawlessly.

Of course now that I say that....

Thanks for the thoughts/interactions Jay and Matt.

--greg

Wow. Last night, in the middle of the night, it started failing again. I tried connecting from both of my LANs, directly on cable, etc. I tried different DNS servers. Nothing—my server was completely unable to reach xxx.indigodomo.net. I was able to reach it from an external server (my university). So, I just tried using a VPN service (encrypt.me) and all works fine, albeit w/ the hit of the VPN overhead. Surely my cable company is not blocking... I'll try calling them ASAP, around work meetings, etc. I don't hold high hopes for even finding someone who might understand...

[matt (support)]

Sounds like the ISP was having routing issues maybe? I would be surprised if they blocked it. I checked our server graphs for the last 24 hours and connections have been steady.

Hi Matt. Interestingly it has also stopped working even when I was using the VPN. It's just set to the "fastest server" (in the US) so I'm not sure where it's going. I'll let you know if/as I make any progress w/ the cable company, just to close the loop on this (if I can). Thanks.

Checking back in on this, right now, I'm getting occasional disconnects at random times. Only one in the last 24 hours. When I had the browser window open, I also got random errors but more frequently.

That said, I have no idea what changed to stop the very consistent disconnects. However, since they don't happen often, I'm going to leave it alone for now.

Thanks for your help.

--Dave

FWIW, my reflector connection started failing again at about 3pm on 3/24. Since then, it's been failing every 2.5 hours as it did before. There was no change before or after that point in the number of ping timeouts (which other people I've talked with also experience), so I still think those two issues are unrelated.

Significantly, the errors do indeed go away when I keep a device folder open in a browser window on the server.

I still think the issue is related to the persistent connection timing out since the timing is very much as described in articles on the issue. I also can't think of any other possible cause for such consistent results. Since web page works, maybe the type of communication used in the messages periodically sent by the server is not working as intended to keep the connection alive.

Not knowing what to look for, I haven't been able to find any sign of periodic communication between the reflector and my server. Can you provide a little more information about what is supposed to be sent? Process, port, message type, or whatever. If you have any suggestions on how to look for the messages, that would be very helpful. Thanks!

--Dave

[matt (support)]

Indigo's keep-alive mechanism is the Indigo Server doing an HTTP request to your reflector every 10 minutes. That request gets sent out from the Indigo Server to our hosted server where, just like any HTTP request to your reflector, it is routed back to your server over the SSH tunnel. If that request fails for any reason you see the error messages shown in your Event Log and Indigo tries to re-establish a new tunnel. There really shouldn't be any difference at the protocol/messaging level as to what is occurring for this keep-alive compared to leaving a browser window open. They both would be making HTTP requests to the same server and sent over the same SSH tunnel. The only different might be in the frequency. Leaving a browser page open will have it check every few seconds versus every 10 minutes for Indigo's internal keep-alive message. I wonder if your router (or an ISP router somewhere in the route) is closing down the connection after just a few (less than 10) minutes of inactivity.

I wonder if your router (or an ISP router somewhere in the route) is closing down the connection after just a few (less than 10) minutes of inactivity.

If that were the case, wouldn't I be getting the reflector error every 10 minutes rather than every 2.5 hours? Can you think of any way I could check for these http messages, like in a log somewhere?

[matt (support)]

Only if we assume the router (or your ISP's routers) are acting consistently. Based on the pattern you've seen, that doesn't appear to be the case. Indigo is definitely sending the internal messages every 10 minutes (and rebuilding the ssh tunnel if it fails) or we would have lots of additional reports of issues. You can test it yourself as well – just disconnect your Mac from the Internet at a random time (turn off the network, unplug your router, etc) and within 10 minutes you'll see Indigo log the errors you've seen and try to re-establish the connection.

You could use something like Wireshark to inspect the low level packets. I'm not very familiar with using it though and I'm not sure it will really reveal much.

The root problem here is that Indigo's reflector tunnel requires a persistent socket connection to be up, and some networking configurations / routers / etc. aren't great about keeping the connections up. Indigo tests for broken connections every 10 minutes but it isn't going to be able to catch connections that go down more frequently than that.

Also going back to your original log, it is showing that sometimes after Indigo detects the connection is down it is unable to re-establish a new connection (and so it has to re-try multiple times). I'm 98% sure this is a networking issue (router problem, ISP issue, etc.). Indigo is retrying and it fails so it tries again and then it eventually works.

welchgregusa: what ISP do you have? Is it Comcast (Xfinity)?

Yes, xfinity. I guess I’m not understanding how it can fail with such 2.5 hour regularity if it’s the result of random connection issues. I won’t bother with this anymore since keeping the web page up fixes the problem. Thanks for trying, Matt.

—Dave

One more potentially important piece of info: For some reason, the ping timeouts on my server have gone away and the reflector failures are always corrected on the first (5 second( retry attempt. I had tried turning on QoS on my router for all ports on my server. but I don't think this is relevant since the ping timeouts went away for my laptop as well.

This tells me that 2 separate issues may be happening: (1) random network errors causing ping failures and problems reconnecting to the reflector and (2) Reflector connection failure consistently every 2.5 hours. I did confirm that the reflector fails after 10 minutes when disconnecting my network from the server so it still appears that the periodic communication sent to to the reflector is not sufficient to act as a "keep-alive" message to maintain the persistent connection alive whereas the messages sent to the browser are sufficient. However, without more insight or control of the keep-alive communication, I don't think I can take this any further.

Hi again Matt,

Sorry, I guess I can't leave this alone after all. I think I found the difference between running the device folder browser page that prevents the errors and the server reflector checks that don't.

I happened to notice that my router log shows the reflector checks sent from my indigo server every 10 minutes. I thought that was correct except that I did not see similar log entries when I run the device folder browser window.

On a hunch I tried changing the browser url to http instead of https and then the accesses showed up in the log every few seconds. This tells me that the messages you send out from the server are http and not https since they show in the log.

My guess is that the router log only shows an entry if it is a new connection. Maybe the https messages keep the same connection active, they don't show in the log.

Does this make sense to you? Are you indeed sending plain http messages to check the reflector connection?

Thanks.

--Dave

[matt (support)]

I guess I’m not understanding how it can fail with such 2.5 hour regularity if it’s the result of random connection issues.

Just a hypothesis, but what if your router or your ISP had some clean-up task that runs every 150 minutes which closes all socket connections that have not had traffic in the last N minutes? (where N is less than the 10 minutes that the Indigo heart-beat is sent)

Point being, I'm 99.9% sure the 2.5 hours isn't coming from Indigo, so that means it is further down the network chain (router, ISP, etc.). Something is prematurely closing the socket connection. It shouldn't be doing that since Indigo forces traffic over the socket every 10 minutes, but sometimes routers and ISPs do things they shouldn't.

Indigo's internal keep-alive is indeed using HTTP. Since leaving a browser window open helps, I suggest you do the following to add your own internal keep-alive message using Indigo:

1) Create a new Schedule. Have it execute every N minutes. I'd try 1 or 2 minutes for starters to see if that prevents the premature socket closing.

2) Select the Hide Executions in Event Log checkbox at the bottom of the Schedule panel UI.

3) Have the Action be to execute the following embedded python script (just paste it in):

Code: Select all

import urllib

query_url = 'http://YOUR_REFLEDCTOR_NAME_HERE.indigodomo.net/prismstatus'
try:
   result = urllib.urlopen(query_url, None)
   if result is not None:
      server_reply_str = result.read()
      if server_reply_str == 'connected':
         pass
         # Comment out or delete this next log line after verifying it is working:
         indigo.server.log('reflector test returned connected')
      else:
         indigo.server.log('reflector test returned ' + server_reply_str, isError=True)
   else:
      indigo.server.log('reflector test returned None', isError=True)
except IOError, e:
   indigo.server.log('reflector test error: ' + str(e), isError=True)

Edit the query_url string at the begging to be your reflector. After you verify it is running every couple of minutes (in the Event Log) you can comment out the line mentioned in the script so that it will only log if there is an error trying to make the HTTP call.