Z-wave (missing or disabled):

Posted on
Thu Oct 25, 2018 7:29 am
neilk online
Posts: 125
Joined: Jul 13, 2015
Location: Reading, UK

Re: Z-wave (missing or disabled):

I have this problem with a Gen 5, both with the original and latest firmware. It got to the point that I couldn't run an optimise without the stick locking up (and only physically unplugging it would resolve it). I manually optimised every device one by one, and did some housekeeping excluding devices that could be unplugged (wall plugs primarily used for Christmas lights) and touch wood that seems to have resolved it. An optimise now completes successfully, I suspect it gets into the state when a lot of activity happens (it also happened occasionally running the night time shut down action group).

Posted on
Thu Oct 25, 2018 10:15 am
matt (support) offline
Site Admin
User avatar
Posts: 18747
Joined: Jan 27, 2003
Location: Texas

Re: Z-wave (missing or disabled):

Patrik wrote:
Its series 2!
Have a gen 5 laying around...But if its the same its not worth to migrate?!

Yeah, since we have reports of the problem with gen 5 as well I'm doubtful migrating to it will help much. They have a newer firmware that just came out, but it isn't clear if that fixes the problem either.

Image

Posted on
Fri Oct 26, 2018 12:24 am
Umtauscher offline
User avatar
Posts: 344
Joined: Oct 03, 2014
Location: Cologne, Germany

Re: Z-wave (missing or disabled):

Just to add my 2 ct.
I never had this problem with Gen2, so I considered this to be a Gen5 problem.
But since others are experiencing this problem with Gen2 also, I'm thinking it's a problem introduced with Indigo 7.1.1
I updated very late to 7.1.1 skipping 7.1 and after that the problems startet.

I never had any stability problems before 7.1.1
Unfortunately I cannot go back, because I have some devices that require the 7.1 z-wave update.
Cheers
Wilhelm

Posted on
Fri Oct 26, 2018 6:57 am
matt (support) offline
Site Admin
User avatar
Posts: 18747
Joined: Jan 27, 2003
Location: Texas

Re: Z-wave (missing or disabled):

It isn't related to any Indigo changes. I'm 98% sure. Indigo has always handled this problem (the Z-Wave controller getting into a bad state and feeding Indigo misinformation about devices) the exact same. We need to improve how Indigo handles the situation, but to be clear the root problem is the Z-Wave controller is giving Indigo misinformation because it is in a bad state. What causes it to be in a bad state? I don't know, but I believe it is more likely to occur the more modules you have.

Image

Posted on
Fri Oct 26, 2018 8:21 am
Umtauscher offline
User avatar
Posts: 344
Joined: Oct 03, 2014
Location: Cologne, Germany

Re: Z-wave (missing or disabled):

Ok..
Additional information:
The "new" firmware doesn't help either.

Posted on
Fri Oct 26, 2018 9:50 am
neilk online
Posts: 125
Joined: Jul 13, 2015
Location: Reading, UK

Re: Z-wave (missing or disabled):

In my case about 40 modules, so I guess that is a reasonable amount.

Posted on
Sat Dec 01, 2018 10:20 am
colin_d offline
Posts: 17
Joined: Nov 08, 2014
Location: East Sussex

Re: Z-wave (missing or disabled):

Six months back my Gen 2 Z-Stick had several episodes of the ‘all devices disabled’ problem, so I purchased a spare stick just to be on the safe side. I made a backup but continued using the old one which then started behaving itself…

About 3 weeks ago however, out of the blue it happened again. It seemed to be provoked on almost any device changes, even doing a single optimise network would kick it off. I copied the script from viewtopic.php?f=58&t=19916&start=30#p163225 into a trigger which helped with re-enabling devices and notifying me, which is great, but if you can’t rely on the devices working then it’s a non-starter.

So when the system seemed relatively stable I backed up the Z-Stick and copied it to the new one waiting on standby. Guess what. Exactly the same problem. So I had just copied corrupt data from one Z-Stick to another. That pretty much rules out a hardware issue.
I had a ‘clean’ Z-Stick backup but it was out of date by now and how can you be sure there isn’t some corruption lurking in there?

Only one thing for it really, factory default the Z-Stick and re-include everything from scratch. Just 44 ZWave devices, 74 if you include the secondary ones. So I did that, excluding then including each device in small batches then re-syncing with the old indigo definitions, took over half a day and an evening to get them back and enabled. At least all the triggers, schedules & actions were still intact. Most devices needed parameters setting though, as if from new. That’s when you’re glad you’ve kept all the manuals :)

Anyway, It’s now 5 days and counting - no problems. I might copy the data back to the old Z-Stick and replace that at some point , I have a feeling it’ll probably work fine. (Glad I didn’t rush to buy a third Z-Stick!).

The question is, how is the data being corrupted in the first place?

I’ll post any developments… :shock:

Posted on
Sat Dec 01, 2018 10:54 am
jay (support) offline
Site Admin
User avatar
Posts: 14894
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Z-wave (missing or disabled):

colin_d wrote:
That pretty much rules out a hardware issue.

[SNIP]

The question is, how is the data being corrupted in the first place?


It's some kind of firmware issue on the stick, so one could say that it's a hardware issue if you include the firmware on the hardware as part of the hardware.

We're only using the standard Z-Wave serial APIs to talk to the stick (any stick that uses the Z-Wave serial API), so we don't think there's anything we're doing that would cause corruption (vs Insteon, where we sometimes have the need to manipulate low-level bits in the hardware itself). I don't know of any other interfaces (besides the Z-Stick) that get into this corrupt condition (if you have examples please post), so that supports the theory that it's a firmware bug in the Z-Stick line.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Sat Dec 01, 2018 11:04 am
matt (support) offline
Site Admin
User avatar
Posts: 18747
Joined: Jan 27, 2003
Location: Texas

Re: Z-wave (missing or disabled):

colin_d wrote:
... It seemed to be provoked on almost any device changes, even doing a single optimise network would kick it off. I copied the script from viewtopic.php?f=58&t=19916&start=30#p163225 into a trigger which helped with re-enabling devices and notifying me, which is great, but if you can’t rely on the devices working then it’s a non-starter.

I've been looking at what Indigo can do to workaround the problem, including not having it disable devices even though the Z-Stick is handing us bad information on most of the nodes. So after running the script to re-enable the devices does everything work okay for a while? Indigo only auto-disables device communication when the interface is restarted (or physically removed/reinserted, or Indigo Server restarts) so I would hope that after running the script the devices work okay; at least until the next server or interface restart. If that isn't the case, then how are things failing specifically?

Image

Posted on
Sat Dec 01, 2018 1:32 pm
siclark offline
Posts: 632
Joined: Jun 13, 2017
Location: UK

Re: Z-wave (missing or disabled):

matt (support) wrote:
? Indigo only auto-disables device communication when the interface is restarted (or physically removed/reinserted, or Indigo Server restarts) so I would hope that after running the script the devices work okay; at least until the next server or interface restart. If that isn't the case, then how are things failing specifically?


Is there a difference between auto disables, and disables? If not Indigo definitely disables my devices without any restart of anything. Unless it restarts the interface by itself?

Posted on
Wed Dec 05, 2018 6:06 pm
colin_d offline
Posts: 17
Joined: Nov 08, 2014
Location: East Sussex

Re: Z-wave (missing or disabled):

matt (support) wrote:
I've been looking at what Indigo can do to workaround the problem, including not having it disable devices even though the Z-Stick is handing us bad information on most of the nodes. So after running the script to re-enable the devices does everything work okay for a while? Indigo only auto-disables device communication when the interface is restarted (or physically removed/reinserted, or Indigo Server restarts) so I would hope that after running the script the devices work okay; at least until the next server or interface restart. If that isn't the case, then how are things failing specifically?

Thanks guys for your quick response. Yes after being re-enabled the devices would work okay, at least until the interface decided to get scrambled again.
For your info it all started with errors like this:
Code: Select all
17 Nov 2018, 16:33:38
   Z-Wave Error                    send getVersion failed after 9 attempts
   Z-Wave Error                    restarting serial connection in 5.0 seconds
   Interface Failure               lost connection with interface

17 Nov 2018, 16:34:20
   Z-Wave                          connected to Z-Wave 3.95 static controller interface on /dev/cu.usbmodem1411 (firmware 1.00)
   Z-Wave Error                    module 007 included in controller no longer matches device "007 - ROUTER POWER" type (controller module type: zwDimmerType/zwDimmerType (4, 17, 1), indigo device type: zwRelayType (4, 16, 1))
Then lots of similar lines with modules not matching devices, followed a little later by:
Code: Select all
17 Nov 2018, 17:39:40
   Z-Wave Error                    serial port communication error: read failed: [Errno 6] Device not configured
   Z-Wave Error                    Z-Wave 3.95 static controller might be disconnected
   Z-Wave Error                    silently retrying serial connection every 5.0 seconds
   Interface Failure               lost connection with interface
   Z-Wave                          connected to Z-Wave 4.54 static controller interface on /dev/cu.usbmodem1411 (firmware 1.01)
   Z-Wave                          found module included in controller with no matching device (missing or disabled): 007 - Relay Power Switch
and so on for the remaining devices.
Anyway, since doing a factory reset on the stick and re-including all the devices it has been fine so lets hope it's solved for the moment. :)
10 days and counting.

A thought occurs to me, maybe in a similar situation if you did a backup of the stick, then factory reset it, then restore the backup (I think it's only an image of the external EEPROM) it would reset the CPU, RAM and firmware in the stick but maybe leave the restored device data intact? If it should happen again I'll try that. Hopefully it won't :?

Posted on
Wed Dec 05, 2018 6:49 pm
lanbrown offline
Posts: 727
Joined: Sep 26, 2017

Re: Z-wave (missing or disabled):

I have two sticks and after I add a few devices I go and pull the working stick, and make a backup copy of it. I keep multiple iterations of the backup. This way if something happens I can always restore on the other and if the backup is corrupted I can go to a previous backup and then just include the devices added since that backup. It only takes a few minutes to perform a backup. This seems like a much better way than hoping than performing a backup and restore will work after it gets into a weird state.

Posted on
Thu Dec 06, 2018 3:46 am
colin_d offline
Posts: 17
Joined: Nov 08, 2014
Location: East Sussex

Re: Z-wave (missing or disabled):

lanbrown wrote:
I have two sticks and after I add a few devices I go and pull the working stick, and make a backup copy of it. I keep multiple iterations of the backup. This way if something happens I can always restore on the other and if the backup is corrupted I can go to a previous backup and then just include the devices added since that backup. It only takes a few minutes to perform a backup. This seems like a much better way than hoping than performing a backup and restore will work after it gets into a weird state.

Agreed. I was doing that, but after a few months of faultless operation I got out of the habit of backing up the stick (oops)!
Then I wasn't sure which might be the best old backup to use as the latest one didn't solve the problem. Hence the reset and re-include.
I'm thinking I should've tried a reset then restore.

The ZM5101 main chip in the Z-Stick has it's own internal Flash memory & RAM for firmware and operation. My thought is that it's the RAM/Firmware that's corrupted and a reset to defaults would clear it. It's powered 24/7 by the internal battery of course
I just wonder if the data could still be okay if then restored (the backed up data just happens to be the same size as the external EEPROM in the Z-stick).
More curiosity than anything, and I'm probably barking up the wrong tree but I'm just trying to understand whats going on.
Hopefully it'll behave itself now and I won't get the opportunity to test my theory! :) :) :)

Posted on
Thu Dec 13, 2018 6:46 pm
matt (support) offline
Site Admin
User avatar
Posts: 18747
Joined: Jan 27, 2003
Location: Texas

Re: Z-wave (missing or disabled):

siclark wrote:
matt (support) wrote:
? Indigo only auto-disables device communication when the interface is restarted (or physically removed/reinserted, or Indigo Server restarts) so I would hope that after running the script the devices work okay; at least until the next server or interface restart. If that isn't the case, then how are things failing specifically?


Is there a difference between auto disables, and disables? If not Indigo definitely disables my devices without any restart of anything. Unless it restarts the interface by itself?

Communication can be manually disabled in the UI, but the auto disables should only occur (I think) when the Z-Wave Controller communication initialization is occurring. So that would be on server restart, Z-Wave interface reload (menu or other), or if there is a Z-Wave USB communication failure bad enough that Indigo forces a reload of the interface.

The next time you notice devices are disabled take a look at the Event Log and see if you can trace back to exactly when the problem occurred (it'll be higher up than the lines logging Indigo disabling devices but include those too) and copy/paste those lines into a reply for me.

Image

Posted on
Thu Dec 13, 2018 6:49 pm
matt (support) offline
Site Admin
User avatar
Posts: 18747
Joined: Jan 27, 2003
Location: Texas

Re: Z-wave (missing or disabled):

colin_d wrote:
matt (support) wrote:
I've been looking at what Indigo can do to workaround the problem, including not having it disable devices even though the Z-Stick is handing us bad information on most of the nodes. So after running the script to re-enable the devices does everything work okay for a while? Indigo only auto-disables device communication when the interface is restarted (or physically removed/reinserted, or Indigo Server restarts) so I would hope that after running the script the devices work okay; at least until the next server or interface restart. If that isn't the case, then how are things failing specifically?

Thanks guys for your quick response. Yes after being re-enabled the devices would work okay, at least until the interface decided to get scrambled again.
For your info it all started with errors like this:

Code: Select all
17 Nov 2018, 16:33:38
   Z-Wave Error                    send getVersion failed after 9 attempts
   Z-Wave Error                    restarting serial connection in 5.0 seconds

Thanks for the additional information. That first log line is occurring when Indigo is initializing communication with the Z-Stick. Can you trace back up higher than that and find out why Indigo is re-initializing communication with the Z-Stick? That should only occur on server restart, interface reload, etc.

Image

Who is online

Users browsing this forum: No registered users and 2 guests