Failure Recovery Strategy Needed

I had a few things happen that are forcing me rethink a strategy regarding recovering from failures.

#1) Yesterday my Mini 'hung' [I can't remember the number of years since that has happened, if ever] and of course it happened during a drip irrigation cycle. The only reason I noticed an issue was that my iPad screen went black, with the message "unable to connect to Indigo server", and my remote client disconnected and would not reconnect.

Thankfully I was at home, but by the time I figured out what was going on and rebooted the Mini my potted plants were aquariums. Had I not been home I would have water front property.

#2) Today I noticed that during a sprinkler cycle

Code: Select all

Schedule                        _next_zone_1_Irrigation [sprinkler]
  Sprinkler                       Turning "Irrigation [sprinkler]" zone 2 on for 28.00 minutes
  Error                           "Irrigation [sprinkler] - [2] Pool Left" on; send failed (no acknowledgment)

The irrigation actually went to zone 2, but all interfaces displayed zone 1, even as zone 3 started. I corrected the issue by using the 'get valve status' button in the client.

I proceeded to create a CP element that I could press to correct the valve display. This is when I discovered that the sprinkler control action includes everything EXCEPT 'get valve status'.

So, who has some ideas? Off the top of my head I would like:
- A CP element to activate 'get valve status' manually.
- A trigger that on every zone change would 'get valve status' after ±15 seconds, healing itself if there were an error.
- A solution for rebooting your server manually and/or automatically remotely. It would be nice if the computer could just sense the problem and reboot itself. The biggest problem with a 'hang' is no alerts are sent so if you were away from home you would have no idea there was a problem like in a power failure.

Indigo 5.1.10
EZ Flora's

[jay (support)]

We don't currently expose UI to send sprinklers a status request (I'll get it in in a dot release after we ship 6.0GM), but you can do it in a Python script:

Code: Select all

indigo.device.statusRequest(SPRINKLER_ID_HERE)

Just substitute the sprinkler's ID and that will get all valve states and status.

You can use the "Error in Event Log" trigger event (use Internal Indigo Errors) in the SQL Logger plugin and any time there's an internal Indigo error you can schedule a delayed status request to the sprinklers. Of course, it's not granular for just that error - any Indigo Server generated error will fire the trigger, but assuming you only have sporadic errors (and you wait before sending the status request so as to avoid adding extra traffic when there's likely some kind of signal issue anyhow) that's not a big deal. If you have a lot of errors then that's a different problem.

There's an option in the Energy Saver System Preference (at least it's there on Lion) that shows under the UPS tab (when you have a UPS connected via USB to your computer) called "Restart automatically if the computer freezes". I've never actually tried it before but it sounds like what you're looking for. I suspect that the UPS has to explicitly support that feature but I also suspect most that have a USB connection do. YMMV.

We don't currently expose UI to send sprinklers a status request (I'll get it in in a dot release after we ship 6.0GM), but you can do it in a Python script:

Code: Select all

indigo.device.statusRequest(SPRINKLER_ID_HERE)

Just substitute the sprinkler's ID and that will get all valve states and status.

You can use the "Error in Event Log" trigger event (use Internal Indigo Errors) in the SQL Logger plugin and any time there's an internal Indigo error you can schedule a delayed status request to the sprinklers. Of course, it's not granular for just that error - any Indigo Server generated error will fire the trigger, but assuming you only have sporadic errors (and you wait before sending the status request so as to avoid adding extra traffic when there's likely some kind of signal issue anyhow) that's not a big deal. If you have a lot of errors then that's a different problem.

There's an option in the Energy Saver System Preference (at least it's there on Lion) that shows under the UPS tab (when you have a UPS connected via USB to your computer) called "Restart automatically if the computer freezes". I've never actually tried it before but it sounds like what you're looking for. I suspect that the UPS has to explicitly support that feature but I also suspect most that have a USB connection do. YMMV.

Thanks for the info especially the "Restart automatically if the computer freezes" pref. I'm still on 10.5, but I have a new mini w/ 10.8 just waiting for Indigo 6 GM.

Unfortunately Apples documents state, "These options aren’t available on all computers", so it sounds like it's more computer model specific than OS version. Hopefully it's there when I plug it in.

Regarding the status request, would this work as a self heal from UI errors [where what's seen in UI doesn't match reality]?

TRIGGER
- active zone name
- has any change

ACTION
- execute script
- python indigo.device.statusRequest(SPRINKLER_ID_HERE)
- delay 15 sec

[jay (support)]

Well, not really, since the failure that you point out is that Indigo doesn't update the active zone even though it did in fact change. That's why I suggest triggering off an error instead.

There's an option in the Energy Saver System Preference (at least it's there on Lion) that shows under the UPS tab (when you have a UPS connected via USB to your computer) called "Restart automatically if the computer freezes". I've never actually tried it before but it sounds like what you're looking for. I suspect that the UPS has to explicitly support that feature but I also suspect most that have a USB connection do. YMMV.

It's NOT there on:
OS 10.8.4
Mini 2.3 i7
Tripplite [smart1500lcdxl] UPS

If any user has this Energy Saver feature using 10.8 please reply with:
Computer model
UPS model
OS 10.8.x or 10.8.x server

[jay (support)]

Just rebooted into Mountain Lion and it looks like Apple removed that option. Maybe it didn't work that well so they just dumped it.

[edit] Just for giggles I rebooted into Mavericks and all the options under UPS are gone. Not sure if that's just because of it being beta or if they're just eliminating UPS integration all together.