upgraded to 2022.2, now get repeated errors "send failed..."

Posted on
Wed Mar 15, 2023 12:26 pm
kyphos offline
Posts: 54
Joined: Dec 17, 2014

upgraded to 2022.2, now get repeated errors "send failed..."

I've recently upgraded my Indigo platform which has been running happily since 2014 or so.
Old: a well-worn Mac mini(2,1) with OSX 10.7.5 (Lion) and Indigo 6.1.11
New: a newer Mac mini (2.3 GHz Intel Core i5) with macOS 10.13.6 (High Sierra) and Indigo 2022.2.
No change: Insteon PowerLinc 2413U modem and USB cable.

Since the migration, I'm finding several errors in the Event log every night at 11:45pm.
The errors are all "send failed (no acknowledgement)", occurring when Indigo sequentially sends OFF messages to various Insteon lights and outlets.

The cluster of errors occur nightly at 11:45pm, triggered by a scheduled event.
That event contains a single action:
Type: All Off (Light Appliance Controls)
Devices: All INSTEON/X10.

I have a few old X-10 devices, and 9 Insteon dimmers, switches, and outlets.
At 11:45pm, the log shows the server sends an X10 command to turn all units off. There's no response nor error since X10 is a one-way protocol.
The server then appears to send sequential OFF messages to the 9 Insteon devices.
Some result in "no acknowledgement" errors. It seems random which devices succeed and which result in an error.

I've done a bunch of manual testing, using both the Indigo Touch app on my iPhone and the Indigo 2022.2 app on my Mac to send ON and OFF commands to individual devices. No errors are generated. But like clockwork, every night the process by which the server executes the All INSTEON - OFF action leads to errors.

This behavior started after migrating to the new Indigo environment. I still have the old Mac mini on hand, and I've inspected the Indigo logs from before it went out of service. No errors were observed in the logs, going back several days..

So what's changed? The Mac mini hardware is new, and faster. Indigo 2022.2 is new. The underlying OS is new (now 10.13.6). And there's no add-on 3rd-party USB driver which was required with 10.7.

The logs also show occasional "send failed (no acknowledgement)" errors during the day, all triggered by various scheduled events. A few per day. I don't see any pattern as to time of day, or device.

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Wed Mar 15, 2023 12:45 pm
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: upgraded to 2022.2, now get repeated errors "send failed

Is you PowerLinc plugged in to the same outlet it was when you were using the old Mac?

It sounds like there are some signal collisions going on because of all the Insteon commands going out at once. This is likely exacerbated because of some change in your power line system that's added some signal attenuation.

In general, trying to send a bunch of Insteon commands all at once can cause collisions (something Insteon can't handle very well). I'd recommend changing from using the all off command for the Insteon devices to individual off commands to each device so that you can add a small delay between some to kinda spread it out over a few seconds.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Wed Mar 15, 2023 1:56 pm
kyphos offline
Posts: 54
Joined: Dec 17, 2014

Re: upgraded to 2022.2, now get repeated errors "send failed

Jay,
There's been no change in the power line wiring or the physical setup. No change in signal attenuation.

I'd guess that the Indigo action to turn off ALL INSTEON devices gets implemented with some sort of a Do/While loop (or For I=1 to N) that sequentially issues individual Insteon commands to each device whose type=Insteon. That sequence will happen faster on new Mac hardware. The USB driver has changed as well (it's now integrated as part of macOS). It could be that the Indigo code is now trying to send commands too often and overrunning a buffer somewhere. Or perhaps the responses from the various devices are coming back too quickly and the USB driver (or the PowerLinc) is suffering congestion. Either flaw would result in the cluster of errors that get logged nightly at 11:45pm.

But it doesn't explain the singleton failures. As noted above, I'm also seeing errors when a scheduled event sends out a command to a single device. The log reports a few of those each day. They can't be due to collisions as there's only one command going out (and one response coming back). This behavior just started -- I did not see any such errors with 6.1.11 on the old Mac mini.

Note that these singletons only seem to occur following scheduled events. I've been watching for errors while sending dozens of Turn ON and Turn OFF commands manually (using the 2022.2 application). Have yet to see a 'send failed' error following a manual command.

I will try what you suggested, and break the 11:45pm scheduled event into a series of individual OFF commands. But that won't have any impact on the various singleton errors that occur occasionally throughout the day.

Will update this with any new information.

Thanks.

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Wed Mar 15, 2023 4:15 pm
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: upgraded to 2022.2, now get repeated errors "send failed

Indigo queues up commands, and when the PowerLinc indicates that the line is clear, we tell it to send the signal. This logic and, in fact the entire Insteon interface (aside from some device definitions) hasn't changed since Indigo 5. Collisions occur when Insteon device send unsolicited messages (change of state, scene broadcasts, etc) at the same time that other devices try to send.

Insteon is extremely sensitive to signal noise. Because you believe nothing on your powerline has changed doesn't mean something hasn't. I've had power strips that I've used for years start to fail and that caused increasing signal noise until I had to do some signal troubleshooting. Unfortunately, this is just the nature of Insteon.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Sun Mar 19, 2023 2:33 pm
kyphos offline
Posts: 54
Joined: Dec 17, 2014

Re: upgraded to 2022.2, now get repeated errors "send failed

Jay,
Thanks for the comments.
I'm well aware of the issues of power-line noise going back to my X-10 days, and my power line infrastructure is pretty well-hardened. My PowerLinc is plugged into a receptacle right next to the breaker panel. It's on one side of a dedicated 240 VAC breaker. There's an X10 bridge circuit between the phases on that breaker to couple the PLC messages from one side to the other. I have some heavy-duty power line filters (big chokes) on the power circuits feeding my IT closet and my home entertainment system, to keep Hi-freq noise at bay. As a result, all my old X-10 and Insteon gear has been working fine, until recently.

It's a real mystery, and I'm baffled. As I wrote earlier, I started noticing multiple Insteon errors nightly at 11:45pm when an end-of-day schedule ran. Since then, I've done a ton of testing, trying to determine if the Insteon messages sent by Indigo were not reaching the end-points, or if the replies were not getting back. Results were inconclusive. Ms. Google found some old threads about issues with the USB driver in the early days of High Sierra, so I focused on that for a while. Changed USB ports & cables, unplugged other USB devices, etc. Eventually, I migrated back to my old Lion-based Mac mini with Indigo v6.1.11. Still saw errors in the log nightly at 11:45pm (so the root cause is not a bug in the macOS USB driver, nor in 2022.2 :) )

So yesterday, I set up another scheduled event that sends OFF messages to a half-dozen Insteon devices. It runs every 20 minutes. All night long, clumps of 'fail to send' errors got logged. I don't see any pattern - some times a few errors, sometimes several. But at 7am, the errors ceased. The test schedule has been triggering every 20 minutes so far today, without a single error logged.

I'm baffled as to what might be the root cause. I've not added any new electrical equipment that might be generating power line noise. No big motors that shutoff before 7am. The furnace is off (until 8am). The occupants aren't awake yet. Perhaps there's a new noisy electrical device in a nearby house that powers down at 7am and comes back to life in the evening, but my neighbors are all a couple hundred feet away so conducted noise from one household to another seems unlikely.

I'm aware of the well-known quality defects in the Indigo PLM design (poor capacitors) and I may try re-capping my 2413U. However, I don't think a failed capacitor would lead to these sort of cyclical errors that come and go.

Will update this thread if I figure out what's going on.

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Sun Mar 19, 2023 5:22 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: upgraded to 2022.2, now get repeated errors "send failed

I don't think this is the problem, but search the Event Log for "sleep" messages coming from macOS and make sure your Energy Saver settings are set to never sleep. If the IndigoServer gets an impending sleep message from macOS, then it has to shutdown its USB communication even if the Mac never goes to sleep. If this is the problem then you'll definitely see some "sleep" related messages in the Event Log. Doubtful that is the problem given the specific errors you are seeing in the Event Log, but worth looking.

Image

Posted on
Mon Mar 20, 2023 10:43 am
kyphos offline
Posts: 54
Joined: Dec 17, 2014

Re: upgraded to 2022.2, now get repeated errors "send failed

Jay,
Thanks for the tip about sleep messages in the event log. I checked - there aren't any. I have the Mac set to Never Sleep (in the Energy Saver pref pane). Been that way since the beginning in 2014.

I observed something last evening that I hadn't noticed before. A lot of my recent testing has been using the Indigo Touch app on my iPhone. I wander around the house clicking lamps on and off to see if they respond. I observe activity in the logs. Etc. But yesterday I was using the Indigo app on the Mac and noticed an anomaly. There's a little progress indicator in the lower-left corner showing the status of the PowerLinc. Normally, after clicking a device on or off, a green activity bar sweeps left-to-right across the indicator, and then a fraction of a second later, changes color and returns the other way. Several times yesterday, I saw the green bar pause for a fraction of a second halfway across as it went from left to right. Then it continued on its way, changed color, and 'bounced back', returning to the left side. A couple of times, the green bar paused for perhaps a second before it made its way across. And on at least one occasion, the pause was longer, 2 maybe 3 seconds. And a 'failed to send - no acknowledgement' error appeared in the log.

Correct me if I'm wrong, but from one of your previous answers I understand the PLM (or perhaps your code) queues up messages to be sent out on the power line until no traffic (or noise) is detected by the modem. So if there's a lot of electrical noise (or spurious X10 or Insteon traffic), the outbound message won't be sent by the PLM immediately. Hence the green bar pauses half-way across, waiting for the line conditions to clear. If/when the noise abates, the message gets sent. And if it doesn't get sent and elicit a reply back from the device within a few seconds, Indigo server then times out and logs an error? Is that how the comms work? If my assumptions above are correct, the slight pause of the green bar is a useful clue as to what's going on.

My testing continues, with ongoing mystery. The test event that I created yesterday continues to send out a half-dozen Insteon commands every 20 minutes. It ran all night without a single error. In fact, the last 'failed to send' error occurred over 12 hours ago. Other than setting up the 20-minute test, I've made no changes. It seems that the source of power-line noise has vanished.

Thanks.

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Tue Mar 28, 2023 12:58 pm
kyphos offline
Posts: 54
Joined: Dec 17, 2014

Re: upgraded to 2022.2, now get repeated errors "send failed

Jay,
FYI - an update, and possible resolution.
Since my first post, I've spent countless hours troubleshooting the repetitive 'failed to send' errors.
One plausible hypothesis, that you suggested, was that something in the house was randomly generating noise on the power line, thus inhibiting transmission or reception of plc messages to/from the 2413 PLM. However, the problem came and went. One night I'd see continuous errors in the log all night long (every 20 minutes when I had a test schedule running). But then they abruptly stopped. I'd see them sometimes during the day - errors for 10/15/20 minutes. And then nothing. I've chased my tail unplugging suspect devices that might be generating power line noise, turning off surge supressors, etc. All to no avail.

Another hypothesis is that the power line noise is originating outside my household. Perhaps a neighbor with a noisy machine that turns on and off. Or a corroded connection out on the power pole somewhere that is occasionally arcing. It's spring time here, and the temperature is going up and down past freezing. I know that freeze/thaw cycles can be a cause of degraded transmission performance of the high-frequency signals used for DSL internet, but I've never heard of such a thing on 240 VAC power. In any event, if the problem I've been seeing is due to power line noise originating from outside, there's not much I can do about it.

The last hypothesis is that my 2413 is the root cause. Insteon's power supply design is well known to be deficient, with cheap capacitors that fail after a few years. It's plausible that capacitor(s) in mine are slowly failing. Not so bad that it's dead, but enough that power line modem circuitry becomes iffy. I'm an EE, and that mode of intermittent failure seems pretty unlikely, but who knows.

So with not much else to try, I decided to replace all the electrolytic caps in my 2413. Since then (3 days ago), I've seen no errors in the Indigo log :D .
Too soon to tell if the 'fail to send' errors have truly been banished, but so far so good.

I just submitted a post on the Insteon section of the forum with details of my 're-capping' project. It may help others hoping to get a few more years out of their Insteon gear.
viewtopic.php?f=7&t=27056

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Tue Mar 28, 2023 1:30 pm
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: upgraded to 2022.2, now get repeated errors "send failed

Cool. It will be interesting to see if this really was the issue.

I'm meeting with someone from Insteon tomorrow and I'll pass along your story.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Mar 28, 2023 2:06 pm
kyphos offline
Posts: 54
Joined: Dec 17, 2014

Re: upgraded to 2022.2, now get repeated errors "send failed

Jay,
Was my analysis of the signal flow in Indigo that I enquired about a couple weeks ago accurate?? You didn't reply.

Correct me if I'm wrong, but from one of your previous answers I understand the PLM (or perhaps your code) queues up messages to be sent out on the power line until no traffic (or noise) is detected by the modem. So if there's a lot of electrical noise (or spurious X10 or Insteon traffic), the outbound message won't be sent by the PLM immediately. Hence the green bar pauses half-way across, waiting for the line conditions to clear. If/when the noise abates, the message gets sent. And if it doesn't get sent and elicit a reply back from the device within a few seconds, Indigo server then times out and logs an error? Is that how the comms work? If my assumptions above are correct, the slight pause of the green bar is a useful clue as to what's going on.


As I've been troubleshooting the last few weeks, I've closely watched the progress of the little green bar across the progress indicator at the lower-left corner.
When everything is working fine, it smoothly bounces back and forth.
Sometimes, it takes a pregnant pause halfway across, then continues. No error is generated.
But sometimes the pregnant pause while it's halfway across extends to a second or two. And then BAM! - an error is logged.

On occasion, I've seen what appears to be two or three little green bars all stalled in the queue. Only for a second or two, before either (a) they get sent out or (b) error(s) are logged.

Thanks.

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Tue Mar 28, 2023 2:30 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: upgraded to 2022.2, now get repeated errors "send failed

kyphos wrote:
Was my analysis of the signal flow in Indigo that I enquired about a couple weeks ago accurate?? You didn't reply.

Correct me if I'm wrong, but from one of your previous answers I understand the PLM (or perhaps your code) queues up messages to be sent out on the power line until no traffic (or noise) is detected by the modem. So if there's a lot of electrical noise (or spurious X10 or Insteon traffic), the outbound message won't be sent by the PLM immediately. Hence the green bar pauses half-way across, waiting for the line conditions to clear. If/when the noise abates, the message gets sent. And if it doesn't get sent and elicit a reply back from the device within a few seconds, Indigo server then times out and logs an error? Is that how the comms work? If my assumptions above are correct, the slight pause of the green bar is a useful clue as to what's going on.

This is fairly accurate. The green bar "nugget" shows as progressing from the left to the middle when Indigo puts it into its internal queue to be sent to the PowerLinc. The queue then builds up (in the middle) waiting for the PowerLinc to report that it is "idle" and there isn't Insteon traffic (or noise) on the power line. Once it reports that Indigo pops a command off the queue and sends it to the PowerLinc. Indigo then shows that nugget progressing from the middle out the right (it is cleared from the progress bar at that point).. If the PowerLinc is then unable to send the command or does not receive back an ACK then an error is logged in the Event Log.

Image

Posted on
Sat Apr 01, 2023 9:12 am
pgershon offline
Posts: 506
Joined: Jul 10, 2004

Re: upgraded to 2022.2, now get repeated errors "send failed

I am having a somewhat similar error and I am going to reactivate my old thread on it, and potentially start a new one off no traction, Since upgrading to 2022.2 less than a week ago, my system keeps getting "Received sleep notification from system" followed almost instantaneously by "Received awake notification from system". Best I can tell, there is nothing provoking the iMac to sleep, nor anything awakening it. Previously over the last few years, there were a few times when the system inexplicably went to sleep, but I would have to do something manually to awaken it. In this case, the iMac goes to sleep and then wakes up, and the only reason I notice is that the MQTT broker plugin stops working and my outside lights dont turn on or off. I will pay more attention over the next few days and weeks to see if I can determine a cause. But it happened yesterday at 1:24 PM while I was in the basement on the Peloton, and then again last night at 1:09 AM.

I do think it relates to 2022.2 because it never happened before. I have all sleep functions on my iMac deactivated (energy saver power nap disabled, put disks to sleep disabled). And I do not believe the machine is actually going to sleep - I think Indigo is just getting and acting on a flag (but I could be wrong since I am never in room with iMac when it happens).

Any ideas?

Posted on
Sat Apr 01, 2023 11:27 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: upgraded to 2022.2, now get repeated errors "send failed


Image

Posted on
Wed Apr 05, 2023 11:51 am
kyphos offline
Posts: 54
Joined: Dec 17, 2014

Re: upgraded to 2022.2, now get repeated errors "send failed

matt (support) wrote:
kyphos wrote:
Was my analysis of the signal flow in Indigo that I enquired about a couple weeks ago accurate?? You didn't reply.

Correct me if I'm wrong, but from one of your previous answers I understand the PLM (or perhaps your code) queues up messages to be sent out on the power line until no traffic (or noise) is detected by the modem. So if there's a lot of electrical noise (or spurious X10 or Insteon traffic), the outbound message won't be sent by the PLM immediately. Hence the green bar pauses half-way across, waiting for the line conditions to clear. If/when the noise abates, the message gets sent. And if it doesn't get sent and elicit a reply back from the device within a few seconds, Indigo server then times out and logs an error? Is that how the comms work? If my assumptions above are correct, the slight pause of the green bar is a useful clue as to what's going on.

This is fairly accurate. The green bar "nugget" shows as progressing from the left to the middle when Indigo puts it into its internal queue to be sent to the PowerLinc. The queue then builds up (in the middle) waiting for the PowerLinc to report that it is "idle" and there isn't Insteon traffic (or noise) on the power line. Once it reports that Indigo pops a command off the queue and sends it to the PowerLinc. Indigo then shows that nugget progressing from the middle out the right (it is cleared from the progress bar at that point).. If the PowerLinc is then unable to send the command or does not receive back an ACK then an error is logged in the Event Log.


Matt,
Wow!! That you spent the time/effort to design and implement the sophisticated real-time progress bar is impressive. The momentary pauses that I've observed from time to time as the green nugget moves left to right corresponds with brief delays in the endpoint going on or off. I now understand a little better what I've been seeing.

But one thing I don't understand is the root cause of the 'backup' in the PLM. Both you and Jay have mentioned that if there's too much noise on the power line, the PLM will hold off on transmitting commands (and thus Indigo will queue up outgoing commands). But it's my understanding that the Insteon protocol is dual-mode, using both power line and RF (@ 915 MHz). If the modem sees too much noise on the power line, can't it send out the command over RF instead? The nearest Insteon device will receive it (if it's close enough), and then repeat it, using both RF and power line signals. That's according to the Insteon docs that I've read.

In any event, my Indigo/Insteon woes have vanished :D :D . As I mentioned a few posts back, I decided to replace all the electrolytic capacitors in my 2413U. The 're-capped' modem was put back into service 10 days ago, and there have been zero 'failed to send' errors since. To be honest, I'm baffled by that result. I well understand why caps fails with old age, and how they can degrade the operation of the power supply. But the nature of the errors I was seeing (rock solid for a while, then numerous errors for hours at a time, then back to error-free operation) is not consistent with a degraded capacitor. However, the errors have vanished, so I've 'closed the case'!

Related note - while I was at it, I decided to recap my 2450 IO-Linc as well. Its power supply is similar to the design in the PLM. Prior to the fix, the LED had been flickering a bit, although functionally the IO-Link was working fine. However, with new capacitors, the flickering has stopped :D

:arrow: Indigo 2023.1.1, macOS 10.13.6 (High Sierra), Mac mini5,1 (Intel Core i5), PowerLinc 2413U

Posted on
Wed Apr 05, 2023 4:36 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: upgraded to 2022.2, now get repeated errors "send failed

kyphos wrote:
Wow!! That you spent the time/effort to design and implement the sophisticated real-time progress bar is impressive. The momentary pauses that I've observed from time to time as the green nugget moves left to right corresponds with brief delays in the endpoint going on or off. I now understand a little better what I've been seeing.

Thanks!

kyphos wrote:
But one thing I don't understand is the root cause of the 'backup' in the PLM. Both you and Jay have mentioned that if there's too much noise on the power line, the PLM will hold off on transmitting commands (and thus Indigo will queue up outgoing commands). But it's my understanding that the Insteon protocol is dual-mode, using both power line and RF (@ 915 MHz). If the modem sees too much noise on the power line, can't it send out the command over RF instead? The nearest Insteon device will receive it (if it's close enough), and then repeat it, using both RF and power line signals.

Unfortunately, that isn't the case. Based on my knowledge and experience, if there is power line noise it prevents the PowerLinc from sending anything even though it can send via both the power line and RF. The "dual-band" advantage might help when there is a signal range issue, but if there is network congestion or noise on just the power line then everything comes to a standstill until it is cleared.

Image

Who is online

Users browsing this forum: No registered users and 4 guests