NOAA Plugin failing

[dtich]

I am, as of today, seeing an odd update failure myself. I have four station devices config'd, usually all work very fine and no issues. Today, none of the stations has updates since 16:40 local; as of this writing that is 7 hours ago. I have checked the Metars and they are all updated since then but the plugin has not updated. Not sure if that means not polled or what, but I've reloaded the plugin a couple times and same behavior. No update on load (I assume it updates on load..?)

This will be an issue very soon for irrigation for me. Anyone else see this behavior? I'm using KBUR, KVNY, etc, for reference.

I suppose next step is totally restart Indigo and see if that breaks it free. Although not clear the issue. I will enable a higher level of debug now.


Thanks.

EDIT: Attaching screenshot, shows log on plugin reload:

[dtich]

Seems to have updated a few mins ago. No idea what would have prevented the update all those hours, all systems were online and no comms issues. Nothing reported in log. I can't seem to find a place where NOAA xml feed system status is reported so.. Not sure how to diagnose, but .. it seems to be back.

Curious if others had that issue.


Thx

[scs]

NOAA only updates the values once an hour. We update them every 30 minutes to at most the data is 30 minutes out of date.

I just got an update for my station. My guess is that your specified station isn't reporting updates to NOAA. That can happen if they have hardware failures, network failures, etc. They are pretty reliable but issues do happen.

I went out to open the chickens this morning and discovered the water heater off once again ....it was 20 degrees out and the water was frozen again.
I whipped out my phone and checked the NOAA Weather in the Indigo App and I said 33 degrees, Obviously incorrect, but that explains why the heater wasn't on.

I had a look at the logs and for the past few days, I can see the NOAA weather updating every 30 mins as below , and then it stopped at 9:16PM last night


Jan 26, 2022 at 8:46:37 PM
NOAA Weather Debug Updating device: NOAA Weather

Jan 26, 2022 at 8:51:20 PM
AD2USB Alarm Interface Zone: 7 - Alarm Zone 7 state changed to: (Faulted)
AD2USB Alarm Interface Zone: 7 - Alarm Zone 7 state changed to: (Clear)

Jan 26, 2022 at 8:52:36 PM
AD2USB Alarm Interface Zone: 7 - Alarm Zone 7 state changed to: (Faulted)
AD2USB Alarm Interface Zone: 7 - Alarm Zone 7 state changed to: (Clear)

Jan 26, 2022 at 9:16:38 PM
NOAA Weather Debug Updating device: NOAA Weather

No further entries from NOAA weather at all.
As if it's hung.
Any ideas things to try? (other than restarting the plugin)

[scs]

Here's what it looked like when I reloaded the plugin :

Jan 27, 2022 at 8:58:34 AM
Reloading plugin "NOAA Weather 2021.1.0"
Stopping plugin "NOAA Weather 2021.1.0" (pid 26109)


Jan 27, 2022 at 8:58:56 AM
Warning process (pid 26109) failed to quit after polite request -- forcing it to quit now
Stopped plugin "NOAA Weather 2021.1.0"
Starting plugin "NOAA Weather 2021.1.0" (pid 30354)
Started plugin "NOAA Weather 2021.1.0"
NOAA Weather Debug Starting device: NOAA Weather
NOAA Weather Debug Updating device: NOAA Weather
Trigger Chick water heat on
NOAA Weather Debug Starting concurrent tread
Sent INSTEON "Chicken Water Heater" on

The fact that it had to be forced down is a bit concerning
You can see that once reloaded , it was able to get an update and triggered the heater to on.
thanks

[dtich]

Mine stopped again in the middle of the night, around 4am, I manually reloaded it and it updated.

Several concerns: when it stops updating it throws no errors, even in debug mode.

Unchecking debug mode does not stop debug logging, even restarting the plugin with the checkbox unchecked hasn't stopped debug entries..I suppose restarting the app will clear it up..?

What do you think the issue is here, Jay? With the stopping updating I mean.


Thx

[jay (support)]

Unclear. Can both of you answer the following:

1) What macOS version are you running?
2) When did you last do an OS upgrade?
3) Confirm the Indigo version you are using.

I need to try to reproduce your environment(s) as closely as possible to try to recreate the problem.

While I'm looking into the problem, you can run a schedule daily that will reload the plugin since a plugin restart seems to have cleared it for you. The plugin id is com.perceptiveautomation.indigoplugin.NOAAWeather

[jay (support)]

Also, please list all the stations you're using.

[dtich]

Thanks Jay.

Indigo 2021.2
MacOS 12.1 (21C52)

KVNY
KBUR
KLAX
KL35
KCQT

OS updated while ago, I recently upgraded all versions from VERY old.. but, was like.. a month ago now. Think OS has stayed the same since, haven't gone to .2 yet. Not in rush to unless there's a reason.

Thx Jay

[scs]

Unclear. Can both of you answer the following:

1) What macOS version are you running?
2) When did you last do an OS upgrade?
3) Confirm the Indigo version you are using.

I need to try to reproduce your environment(s) as closely as possible to try to recreate the problem.

While I'm looking into the problem, you can run a schedule daily that will reload the plugin since a plugin restart seems to have cleared it for you. The plugin id is com.perceptiveautomation.indigoplugin.NOAAWeather

Thanks Jay,

Im running MacOS 10.15.7 with Indigo 2021.2
Ive been running Catalina for approximately a year and a half, but update 7 was done in late November (2 months ago)
I updated Indigo Jan 7 (20 days ago)
I only use KRDU
....I only use the plugin for one thing, and that's turning on the heater when it's below freezing , so I wouldn't tend to notice if it wasn't working until it gets cold here ...that doesn't happen frequently until January.

Rather than reload the plugin every day, what I'd like to do is use another station(KJNX) from 15 miles away, and see if it behaves better.
In the meantime, I have a question, how is the plugin supposed to behave if it fails to connect (for any reason) with the station ?
I'm only asking that in the event there's something wrong at KRDU, I have no reason so suspect the gigabit fiber I use to connect to the internet
thanks in advance
SS

[jay (support)]

Rather than reload the plugin every day, what I'd like to do is use another station(KJNX) from 15 miles away, and see if it behaves better.

As you like.

In the meantime, I have a question, how is the plugin supposed to behave if it fails to connect (for any reason) with the station ?
I'm only asking that in the event there's something wrong at KRDU, I have no reason so suspect the gigabit fiber I use to connect to the internet
thanks in advance

There are a couple of possible errors depending on the failure type. They would start with one of these:

• HTTP error getting station STATIONID data:
• Unknown error getting station STATIONID data:

The line you listed above (NOAA Weather Debug Updating device: NOAA Weather) is all that will show if the call to NOAA is successful, so the fact that it repeats in your log indicates that the network call was successful.

Note that most failures on the NOAA side are data update errors: where their data doesn't get updated, so we just get the same data over and over. We do not update the device with identical data intentionally, so that the last updated date on the device indicates the last time we got new data from NOAA.

[jay (support)]

KVNY
KBUR
KLAX
KL35
KCQT

Well, I think I've identified part of your issue: those stations are pretty darn unreliable in terms of reporting new data. As of right now, here are the last updates from NOAA:

• KCQT: Thu, 27 Jan 2022 09:52:00 -0800
• KL35: Thu, 27 Jan 2022 10:55:00 -0800
• KLAX: Thu, 27 Jan 2022 10:53:00 -0800
• KBUR: Thu, 27 Jan 2022 10:53:00 -0800
• KVNY: Thu, 27 Jan 2022 10:51:00 -0800

So, they all are about 2 hours behind. Who knows how far behind they can get, that's just the spot check as of right now.

Anyway, see my post above: look to see if you see any of the errors indicated or if you see the update line every 30 minutes or not. BTW, I'm not suggesting that there's not an issue with the plugin, only that the data seems pretty old to me for this spot check.

[jay (support)]

Guys, email me a copy of your log files (click on the Show Event Logs Folder and then in the resulting Finder window open the indigoplugin.NOAAWeather folder and send me the files for the days where there was a failure) - send to support@indigodomo.com.

[dtich]

Hilarious isn't it that weather stations located in Class B and C airspaces are so unreliable, lol, oh America... Also, they all have the "$" character in the metar meaning "equipment needs repair or update" lol.

That said, NORMALLY, they are quite reliable and update hourly as expected. The last couple days perhaps not. A cursory glance at the plugin log seems to show that it is indeed polling regularly and updating without error, but the station itself didn't have any updated info. So. Not sure how to take that. See if it looks that way to you.

Thanks!

[jay (support)]

So, yeah - the logs do indeed show that the plugin is getting data from NOAA. The plugin compares the "observation_time_rfc822" field from the XML to what we last got from NOAA - and if they are the same we skip updating any fields. Now, I suppose it's possible that they are updating their data but not updating the observation time, but my spot checks today don't seem to support that. I've been getting updates every hour, though the data itself seems to be approximately 2 hours old.

At this point, I have to think that what you (@dtich) have been seeing is just a failure of NOAA to update its data. And, it was simply coincidence that the data got updated when you restarted the plugin. That seemed like a stretch when I started looking at it this morning, but I can't see any other explanation. If we had any issues talking to the NOAA API, we would definitely have seen it in the logs.

I'll wait to make any conclusions until I get log files from @scs as his issue seems to be slightly different.

For the next release of Indigo, I've added more logging, both informational and debug, that will help us diagnose these types of issues a bit quicker.

I would, however, consider alternatives for outside temperature if it's a critical need - we know that NOAA data can be spotty. I'd recommend some kind of local hardware-based solution so that you are in control of the actual data collection.

[dtich]

Yeah, copy all that Jay. Thanks for looking.

I used to have a weather station here but.. over the years it just phased itself out of the loop as NOAA and Dark Sky were so available and API'able... might have to look into it again.


Thx