Frequent reflector connection errors

Posted on
Sun Mar 08, 2020 11:08 am
DVDDave offline
Posts: 386
Joined: Feb 26, 2006
Location: San Jose, CA

Frequent reflector connection errors

For a while now (months?) I've been getting reflector connection errors like this:
Code: Select all
2020-03-08 09:25:46.822   Error   reflector connection test failed: unable to connect to server
2020-03-08 09:25:46.822   Error   reflector reconnection scheduled in 2 minutes


Recently, they've been happening about every couple of hours and lasting for a few minutes or so, during which time I can't reach my server via the reflector. Internal connections to the server are unaffected and no other network issues are evident. I have a hardwired ethernet connection from my router to my server.

MacOS 10.14.4 with Indigo 7.4.1

Any thoughts on what may be causing this?

Thanks!

--Dave

Posted on
Wed Mar 11, 2020 3:19 pm
matt (support) offline
Site Admin
User avatar
Posts: 20538
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

Hi Dave,

We haven't had any other reports of this and from our server side analytics and charts everything looks good and steady with the overall reflector connection count.

Normally when this type of problem comes up it is either an ISP issue or a router issue. The secure reflector tunnel from your Mac to our server is a persistent TCP connection and sometimes routers will drop/close persistent connections. ISPs (or any of their routers) can also cause the problem. Indigo periodically checks to see if it can round trip a request and when it cannot it rebuilds the tunnel. Do you see errors in the Event Log that the re-connections are failing? Or do the failures always look like above?

Also the next time it is down you can try accessing your reflector from a browser. If you see the "Reflector connection is currently down." message in the browser then the connection from that Mac to our hosted server is good/reachable but the reflector tunnel is down (and should be re-established within a couple of minutes by Indigo).

Image

Posted on
Wed Mar 11, 2020 4:20 pm
DVDDave offline
Posts: 386
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

Hi Matt,

The full set of errors often looks like this:
Code: Select all
2020-03-11 01:40:31.225   Error   reflector connection test failed: unable to connect to server
2020-03-11 01:40:31.225   Error   reflector reconnection scheduled in 5 seconds
2020-03-11 01:40:36.890   Reflector   restarting reflector connection to https://***.indigodomo.net
2020-03-11 01:41:28.099   Error   reflector connection test failed: unable to connect to server
2020-03-11 01:41:28.100   Error   reflector reconnection scheduled in 2 minutes
2020-03-11 01:43:28.856   Reflector   restarting reflector connection to https://***.indigodomo.net

although about 60% of the time it starts back up without the 2 minute wait. It seems to happen about 9 times a day.

When I try from a browser, I do indeed get the "Reflector connection is currently down." message. I guess the reason it often stays down for a couple of minutes is that Indigo waits that long before trying again even though the Mac is connected. Is there any way I can force Indigo to try to reestablish the connection more quickly?

Since all else seems very solid with my Netgear XR500 router, I don't see any settings that I can adjust. Can you think of any settings that may help? Maybe QoS?

Thanks!

--Dave

Posted on
Wed Mar 11, 2020 4:38 pm
matt (support) offline
Site Admin
User avatar
Posts: 20538
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

That failure hints that there is a connectivity problem between your Mac and our hosted server (periodically). Otherwise, we would see the test failing then the reconnection 5 seconds later and no errors for a while. Instead in this case the very next test also fails, so Indigo backs-off on how frequently it will retry the connection a bit (from 5 seconds to 2 minutes). Point being, there is a period illustrated by that log snippet where communication between the Mac and our hosted server isn't working and that it likely isn't just a dropped connection from the router. If during that exact 10 second window you were to try to connect to your reflector in a browser I think you would get an error (not just the "Reflector connection is currently down" message).

So I don't think it is a router setting, but the setting I was referring to is about how the router handles connections. Some routers have a maximum connection table size, or periodically reset/shutdown connections. If you see anything like that in the settings you might try tweaking it (even though I'm not sure it will help in this case). Also try rebooting the router if you haven't. I assume these outages don't coincide with a new dynamic IP address being assigned to your router (by your ISP)?

I'm not sure what the problem is here. Unfortunately these transient networking issues can be difficult to troubleshoot.

Image

Posted on
Wed Mar 11, 2020 5:03 pm
DVDDave offline
Posts: 386
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

Reading more about persistent TCP connections, it seems that they last about 2 hours unless a keep-alive message is sent out from either side of the connection. Since that is about the interval at which my reflector errors occur, this seems relevant. Does Indigo or the reflector server send a keep-alive periodically?

Posted on
Wed Mar 11, 2020 8:00 pm
welchgregusa offline
Posts: 79
Joined: May 29, 2013
Location: Orlando, FL

Re: Frequent reflector connection errors

I am also recently experiencing intermittent (but relatively frequent) errors. I've re-activate my reflector multiple times, rebooted my router multiple time, rebooted the server running indigo, etc. I've seen it become inaccessibly maybe 6 times in the past few days. It seems that re-activating makes it work, for a while, then "Reflector Connection Down." I'm not having any other network issues that I'm aware of.

Posted on
Thu Mar 12, 2020 9:46 am
matt (support) offline
Site Admin
User avatar
Posts: 20538
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

DVDDave wrote:
Does Indigo or the reflector server send a keep-alive periodically?

Indigo forces data over the tunnel much more frequently than this with its own checks to see if the tunnel is up. That is, Indigo has its own keepalive mechanism higher up in the stack that keeps the connection from going stale.

Image

Posted on
Thu Mar 12, 2020 9:54 am
DVDDave offline
Posts: 386
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

Understood. I can't think of anything else that would cause disconnects every couple of hours. I guess I'll just live with it. Not a huge deal although it would be nice to be able to adjust the retry time so I could wait less than 2 minutes.. I'll also look back at old logs to try to determine when the problem began happening. Thanks, Matt.

--Dave

Posted on
Thu Mar 12, 2020 10:00 am
matt (support) offline
Site Admin
User avatar
Posts: 20538
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

A couple of thoughts. First is it exactly 2 hours consistently? What is the variance? Also open a browser window on your Indigo Server Mac and point it to the main (home) page of your reflector (so use your reflector URL and not your Mac IP address) then select a device folder and leave that window open and running. The browser will make a request to your Indigo Server over the reflector (round tripping from your Mac -> reflector server -> back to Indigo no your Mac) every couple of seconds to check for device state updates. I'm curious if that helps to keep the connection alive or not.

Image

Posted on
Thu Mar 12, 2020 10:06 am
welchgregusa offline
Posts: 79
Joined: May 29, 2013
Location: Orlando, FL

Re: Frequent reflector connection errors

Hi Matt. When you say "main (home) page of your reflector" do you mean this one (screen shot attached)....?

Posted on
Thu Mar 12, 2020 10:35 am
DVDDave offline
Posts: 386
Joined: Feb 26, 2006
Location: San Jose, CA

Re: Frequent reflector connection errors

Except for a few outliers, the time between errors varies from 2.3 to 2.8 hours, with the vast majority being 2.4-2.5 hours.

About 1/3 of the time, it comes back in a few seconds on the first retry attempt with the rest coming back after 2 minutes.

I'll try the other test with a browser on my server and let you know what happens.

Thanks for sticking with this, Matt.

--Dave

Posted on
Thu Mar 12, 2020 11:00 am
jay (support) offline
Site Admin
User avatar
Posts: 16767
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Frequent reflector connection errors

welchgregusa wrote:
Hi Matt. When you say "main (home) page of your reflector" do you mean this one (screen shot attached)....?


No. He means open Safari, navigate to your reflector (https://YOURREFLECTOR.indigodomo.net), click on Devices then click on one of your device folders (assuming you have one). You'll want to do this when your reflector is up (it appears to be down given your screenshot).

[EDIT] - sorry, guess No wasn't exactly correct since you have the browser open in the background...

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Mar 12, 2020 1:05 pm
matt (support) offline
Site Admin
User avatar
Posts: 20538
Joined: Jan 27, 2003
Location: Texas

Re: Frequent reflector connection errors

Note I deleted your screenshot since the Start Local Server dialog shows both your reflector name and your username (leaving just the password to be guessed).

Image

Posted on
Thu Mar 12, 2020 2:55 pm
welchgregusa offline
Posts: 79
Joined: May 29, 2013
Location: Orlando, FL

Re: Frequent reflector connection errors

Thanks Matt. Hard to draw conclusions yet but it seems. Perhaps, that having the device folder window open has helped. Or come thing else happened. Or I just haven’t noticed. ( I haven’t noticed it go down since this morning.) Hmm...perhaps I can adjust Dejal to send me a message if it goes down.

Thanks for following alone guys.

Posted on
Fri Mar 13, 2020 4:03 am
welchgregusa offline
Posts: 79
Joined: May 29, 2013
Location: Orlando, FL

Re: Frequent reflector connection errors

The problem persists for me---having the device folder open did not seem to help.

Here are the log entries from 20200312:

https://www.dropbox.com/s/pcncwlh8n91bech/reflector_20200312.txt?dl=0

I tried using the alternate ports, and did not see a change in behavior.

Who is online

Users browsing this forum: No registered users and 2 guests