Typical signal noise issue or time for a tinfoil hat?

Posted on
Wed Jun 02, 2021 9:57 pm
alpitmaster offline
Posts: 13
Joined: Apr 08, 2017
Location: Birmingham, AL

Typical signal noise issue or time for a tinfoil hat?

I am trying to get my arms around a latency problem, but I think it is not the typical type of latency that occurs randomly between switches and loads or between switches and other linked switches. Controller-to-responder latency in my Insteon network is rare, and typically brief when it occurs. I would say that responses to physical switches are near immediate over 95% of the time, even between linked switches, and no more than 3-4 seconds the other 5 % of the time.
After 3 days of troubleshooting, my latency problem appears to be related only to commands that are issued by indigo. And it presents as if Indigo goes to sleep for exactly 25 seconds, during which commands issued by indigo seem to stay in a state of suspension, and then after the 25 seconds is up, Indigo will execute the commands that were issued during the 25 second “sleep” period. It goes to sleep at 1 minute intervals, and oddly enough, it is exactly on the minute, based on the National Bureau of Standards time. I have a watch that updates its time daily by radio, and have confirmed that this “sleep” cycle happens on the minute, every minute.
I have had trigger reliability issues for quite some time (although the occurrences seemed to be random), but I decided to postpone serious troubleshooting until I replaced my aging Late-2012 mac mini, which I did last weekend.
Here’s an example of what I have seen in testing, over the past 3 days:
• I can toggle a light switch on and off repeatedly (one toggle about every 1 ½ to 2 seconds) from the Indigo UI device list, and the light will respond with no delay until the sleep period starts at the top of the minute. “Sent INSTEON” entries appear in the log immediately, each time I toggle the switch.
• At the top of the minute, the light will quit responding to the toggles and no entries will show up in the log until 25 seconds after the minute.
• If I keep toggling the switch after the light quits turning on and off, those extra toggles will get executed, beginning at 25 seconds after the minute, once the software seems to “wake up.” Corresponding log entries will appear as each command is executed.
• At one point, before I began formal testing, I would occasionally get “Error “[device] off [or on]; send failed (no acknowledgement). Before I began troubleshooting, I did a batch sync, and I have not gotten the error messages with the switches I have tested.
• If, during the 25 second sleep period, I toggle a light switch on/off from a linked keypad, the light toggles without delay and a “Received INSTEON” log entry appears with each toggle. I think this tells me that I’m not dealing with a typical signal noise issue. But, I'm looking for an outside opinion.
• I can repeat this any time of the day, with the same results every time.

NOTES ABOUT MY SYSTEM
I am running Indigo on an 8GB mac mini m1, plugged into a dedicated circuit about 15 ft from a subpanel. My 2413u interface is plugged into a separate dedicated circuit. The problems I am describing began some time back, when I was running the Late 2012 mini, so I waited until I replaced the mini to begin troubleshooting.
I have a large Insteon system with 71 switches (I can only imagine how many collisions take place when a command is executed) and a multitude of sensors in and around a 7500 sqft house. Most of the switches are in the 5000 sqft main house, with only a few in the 2500 sqft apartment under the same roof. The sensors are both inside and outside.

QUESTIONS
Of course, my primary question is whether someone can offer any suggestions regarding what is causing this problem. But I would also like to know the timing of when events hit the log. For example when Indigo issues a command, does it get logged immediately, or when it hits the interface, or when an acknowledgement arrives at the interface from the target device? This would help me differentiate a delay in the software execution versus out in the Insteon network.

Posted on
Thu Jun 03, 2021 6:37 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Typical signal noise issue or time for a tinfoil hat?

A typical path would to switch off any plugin in indigo, any trigger , any scheduled action…
If improved: switch back one by one

If not …:?


Sent from my iPhone using Tapatalk

Posted on
Thu Jun 03, 2021 3:10 pm
alpitmaster offline
Posts: 13
Joined: Apr 08, 2017
Location: Birmingham, AL

Re: Typical signal noise issue or time for a tinfoil hat?

A typical path would to switch off any plugin in indigo, any trigger , any scheduled action…
If improved: switch back one by one

Thanks for the suggestion, kw123. I disabled plugins, triggers, and schedules, and rebooted.

After reboot, the "sleep" periods became totally random for about 10 minutes of testing. Then the sleep periods became cyclical--beginning on the minute, lasting for 25 seconds, and reverting to active for the remainder of the minute.

I'm now going to re- post to the Indigo Server Software forum, which I think might be more appropriate.

Posted on
Fri Jun 04, 2021 8:19 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Typical signal noise issue or time for a tinfoil hat?

One more test.
Apparently a restart of indigo changed it. I would do that again and see if you see the same effect


Sent from my iPhone using Tapatalk

Posted on
Fri Jun 04, 2021 1:27 pm
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Typical signal noise issue or time for a tinfoil hat?

The restart changing the behavior, if only for a while, does appear to indicate something systemic. Just to make sure we're on the same page, have you:

  1. Disabled all plugins
  2. Disabled all schedules
  3. Disabled any scripts that may be running
  4. Disabled anything outside of Indigo which may be hitting Indigo (anything using the REST API for example)?

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Fri Jun 04, 2021 2:47 pm
alpitmaster offline
Posts: 13
Joined: Apr 08, 2017
Location: Birmingham, AL

Re: Typical signal noise issue or time for a tinfoil hat?

Thank you for the very prompt response!

I have disabled all schedules and all of my plugins. I did notice these lines when Indigo Server starts: "Loading plugin Action Collection 7.5.0" and
"Starting plugin Action Collection 7.5.0 (pid 608)". I assume that is an internal plugin.
I do not have any scripts of my own. I do all my automation through the Indigo UI.
As far as I can think of, I don't have anything outside hitting Indigo. I have Nest, DSC, and Blue Iris plugins, but they are disabled. If I think of anything else, I'll let message you back.

Posted on
Fri Jun 04, 2021 3:27 pm
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Typical signal noise issue or time for a tinfoil hat?

Add a new schedule (click the Hide executions in Event Log checkbox so that we won't see the actual schedule name in the event log) that fires every 5 seconds with the following script as it's action:

Code: Select all
from datetime import datetime
indigo.server.log(str(datetime.now()))


Let that schedule run for a couple of minutes. What I'm looking for is to see if they stream continuously through the log or if they seem to also get stuck. Copy/paste the results inside code tags in a reply.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Fri Jun 04, 2021 8:33 pm
alpitmaster offline
Posts: 13
Joined: Apr 08, 2017
Location: Birmingham, AL

Re: Typical signal noise issue or time for a tinfoil hat?

The result of your test is below.

I noticed something this afternoon that might help. I’ve never seen an explanation for the progress bar next to the label “PowerLinc 2413u” but I would interpret it that every time a command goes to the interface, it is represented by a green box. They usually exit immediately but I have seen them accumulate in the middle. When a box exits the bar, I’m guessing the interface has passed it out to the network. When a purple box comes in from the right and exits left, I’m guessing that is an acknowledgement. Did I get that right?

What I noticed this afternoon is that when the server is idle, exactly on the minute, the progress bar fills up from the left with 4 green boxes. which gather in the middle. The boxes then exit to the right, one at a time, at approximately 6 second intervals, so that by the time the last one leaves the bar, it is 25 seconds after the minute. That is the 25 second window during which I can’t get actions executed. If I toggle a light on and off, while the 25 second window is closed each toggle will add to the boxes in the middle of the progress bar, and will then release once the window opens again, at 25 seconds after the minute.

Thanks again for your help.
George

Code: Select all
Jun 4, 2021 at 8:41:15 PM
   Embedded script executor host1 started
   Script                          2021-06-04 20:41:15.568241
   Script                          2021-06-04 20:41:20.336096
   Script                          2021-06-04 20:41:25.336476
   Script                          2021-06-04 20:41:30.335914
   Script                          2021-06-04 20:41:35.336030
   Script                          2021-06-04 20:41:40.336155
   Script                          2021-06-04 20:41:45.335918
   Script                          2021-06-04 20:41:50.333208
   Script                          2021-06-04 20:41:55.335874
   Script                          2021-06-04 20:42:00.337595
   Script                          2021-06-04 20:42:05.335812
   Script                          2021-06-04 20:42:10.333893
   Script                          2021-06-04 20:42:15.335831
   Script                          2021-06-04 20:42:20.335499
   Script                          2021-06-04 20:42:25.335739
   Script                          2021-06-04 20:42:30.335473
   Script                          2021-06-04 20:42:35.335941
   Script                          2021-06-04 20:42:40.335888
   Script                          2021-06-04 20:42:45.335731
   Script                          2021-06-04 20:42:50.332757
   Script                          2021-06-04 20:42:55.335483
   Script                          2021-06-04 20:43:00.337142
   Script                          2021-06-04 20:43:05.335401
   Script                          2021-06-04 20:43:10.335501
   Script                          2021-06-04 20:43:15.335375
   Script                          2021-06-04 20:43:20.335138
   Script                          2021-06-04 20:43:25.336081
   Script                          2021-06-04 20:43:30.335188
   Script                          2021-06-04 20:43:35.333267
   Script                          2021-06-04 20:43:40.335459
   Script                          2021-06-04 20:43:45.335174
   Script                          2021-06-04 20:43:50.332399
   Script                          2021-06-04 20:43:55.335062
   Script                          2021-06-04 20:44:00.336662
   Script                          2021-06-04 20:44:05.335106
   Script                          2021-06-04 20:44:10.335172
   Script                          2021-06-04 20:44:15.335083
   Script                          2021-06-04 20:44:20.332492
   Script                          2021-06-04 20:44:25.331948
   Script                          2021-06-04 20:44:30.331865
   Script                          2021-06-04 20:44:35.332077

Posted on
Sat Jun 05, 2021 10:52 am
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Typical signal noise issue or time for a tinfoil hat?

Do you have Z-Wave enabled as well as Insteon?

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Sat Jun 05, 2021 8:51 pm
alpitmaster offline
Posts: 13
Joined: Apr 08, 2017
Location: Birmingham, AL

Re: Typical signal noise issue or time for a tinfoil hat?

No, I have the Z-Wave interface and several sensors, but had decided to wait until I had this problem solved, before adding the other interface.

I'll be glad to install now, if you need me to.

Posted on
Sun Jun 06, 2021 12:27 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Typical signal noise issue or time for a tinfoil hat?

Copy/paste more snippets from your Event Log showing what is logged both when it is working and when it is not.

Also you mention "sleeping" a lot in your post – is there anything in the Event Log about sleep requests from the system? If your Mac is trying to go to sleep then Indigo disconnects communication from the interface (in preparation for the sleep), and then reestablishes communication with the PowerLinc after it gets the wake command. If your system is having problems it might be sending sleep messages to Indigo even though it isn't sleeping. Indigo logs all of this, so check the log for details.

Image

Posted on
Tue Jun 08, 2021 5:07 pm
alpitmaster offline
Posts: 13
Joined: Apr 08, 2017
Location: Birmingham, AL

Re: Typical signal noise issue or time for a tinfoil hat?

Sorry for the delay, Matt. I'm covered up with work, this week, and grabbing a few minutes here and there to troubleshoot, when I can.

I'm going to include an event log snippet with description in this post, and suggest that you guys may want to wait until I can provide you more information, before you put any more time into this. I am seeing instances of longer latency, and more failures to receive acknowledgment than usual. So, I want to try to determine whether there is something in my environment causing that.

However, answering my question below about the status bar will help me separate a potential system issue from a potential environmental problem.

Here is a test I ran yesterday evening.
- Starting at 8:48:35 PM, I toggled a light switch 10 times, finishing at about 8:48:50 PM.
- Each time I toggled the switch, it showed up in the log as it switched the light on or off.
- Then, beginning at 8:49:05, I toggled the switch 6 times, finishing at 8:49:15.
- These actions did not turn the light on and off and did not show up in the log as I executed them.
- Starting at 8:49:28, they finally executed, turning the light on and off, and they appeared in the log as they executed.

Jun 7, 2021 at 8:48:35 PM

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off



Jun 7, 2021 at 8:49:28 PM

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off

Sent INSTEON "5.1.MainHouse--Playroom Long Side" on to 100

Sent INSTEON "5.1.MainHouse--Playroom Long Side" off



In answer to your question about my use of the term "sleep," I'm not referring to sleep, in the sense of the Mac going to sleep. I have the Mac set so that it never goes to sleep, and its disk drive never goes to sleep. Instead, when I speak of Indigo sleeping, I'm referring to the time from the top of each minute until 25 seconds after the minute when, consistently, commands entered from the device list in the UI do not execute and do not show up in the log until approximately 25 seconds after the minute.

I am going to try to include a very short video I shot to illustrate what I'm seeing during that 25 second period. Exactly on the minute, the status bar fills up from the left with 4 green boxes. The boxes then exit to the right, one at a time, at approximately 6 second intervals, so that by the time the last one leaves the bar, it is 25 seconds after the minute. Also, the activity I see in the status bar during that 25 seconds never shows up in the event log. That is the 25 second window during which I can’t get actions executed, and I have been referring to sleeping. It's acting as if it's busy doing something else.

I couldn't figure out how to get a video clip attached, so here's a dropbox link to the 31 second video: https://www.dropbox.com/s/76t7fuje369i8 ... 0.mov?dl=0

Can you tell me what is going on during that 25 second period, And whether or not it is unique to my installation?

Thanks, again.

Posted on
Tue Jun 08, 2021 7:41 pm
jltnol offline
Posts: 989
Joined: Oct 15, 2013

Re: Typical signal noise issue or time for a tinfoil hat?

People much, much smarter than me will help you find the solution.

Things you have probably already tried, but worth mentioning:

Unplug any and all USB devices except for the modem to see if that makes any difference
Quit all programs leaving only Indigo running
Monkey around with the Date & Time settings... disable Set Automatically, and/or change time servers
Disconnect your computer from the internet to see if that changes anything.
Although a little complicated, you might try running Indigo from a different user space on your computer. This would rule out some rouge process running in your regular space.
I can never make heads or tails of the stuff in Console, but might be worth a look at the logs to see what else is happening during this :25 "time out" period.

Like I said, smarter folks than me will solve this, but it just seems that your Mac is getting busy for those 25 seconds doing something that affects indigo's ability to communicate with the modem. So Indigo is holding those commands until communications are reestablished, and then sending them out.

And of course, the other obvious option is something may be wrong with your modem. I've had some success with unplugging mine over night. If you try this, don't forget to unplug the USB connection as well.

Posted on
Sat Jun 12, 2021 7:01 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Typical signal noise issue or time for a tinfoil hat?

alpitmaster wrote:
Can you tell me what is going on during that 25 second period, And whether or not it is unique to my installation?

The green "nuggets" getting stuck midway in the progress bar means that Indigo is waiting for the PowerLinc to report it is idle or available to receive the next command. So the green nugget first appearing on the left means Indigo is adding it to the queue to be sent, then visually is shows them collecting in the middle as it waits for the PowerLinc to get to a state where it can accept them, then finally Indigo sends them out (indicated by the nugget moving from the middle out to the right. If the PowerLinc is idle and not detecting any PowerLinc / RF command or interference then the nuggets will visually not stick in the middle at all.

If the PowerLinc is detecting power line or RF commands or noise then it will not accept the command. It could also be malfunctioning. To try to rule out noise you should plug the PowerLinc into an extension cord (so you don't have to move your Mac), then plug the PowerLinc into different outlets on different circuits throughout the house. Since the problem occurs every minute you should be able to determine if this helps pretty quickly. If it does then you can go on the hunt for the problem circuit / device. You can also try turning off circuit breakers. An extreme example would be to shut off all circuits but your Mac and then see if the PowerLinc can control an appliance or lamp module plugged into that same circuit. Note the problem could be any electronics / bulbs / loads, or could be a malfunctioning Insteon module. It could also be RF interference (not power line). The fact that this happens exactly on the minute makes me think it could be coming from some connected HA device, smart meter, etc. Basically, something with enough smarts to be connecting to a time server.

If none of that prevents the 25 second pauses, then I think your PowerLinc is malfunctioning. I haven't heard of one failing like this before and it would be odd for the failures to occur exactly on the minute given the PowerLinc has no internal clock, but I still wouldn't totally rule out a malfunctioning PowerLinc.

Image

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 2 guests

cron