NOAA Plugin failing

[forestfield]

That 24 hour rollover caught me out badly recently.

Rather than using

Code: Select all

if timedelta.seconds> 7200: 

to get the age, I'd suggest

Code: Select all

 if timedelta.totalseconds() > 7200:

instead. This will give you what I think you want in this case.

-- Paul --

[dtich]

ah, good tip, thx paul.

hey @jay, another anomaly... this am i wake to find the system on the third fallback airport, the first two being not updated in hours, since 224am local.

but when i check metars reports exist for every hour... even after reloading plugin no update. to what do we attribute this?

[jay (support)]

I don't know what the metars report is so I can't really comment.

Here's how the plugin works:

1. We hit the URL for the station selected on the NOAA website: http://www.weather.gov/xml/current_obs/STATIONID.xml
2. We retrieve the "observation_time_rfc822" field from the returned XML
3. We compare it to the last one we got which is saved in your devices "observationDate" state
4. If it's not the same (a simple string compare, nothing fancy with dates), we update the state values

It's actually very simple without much logic. If what we get in the XML from NOAA has any change to the observation_time_rfc822, no matter what the change is, we update the fields with the rest of the data from returned XML. Beyond that I don't know what would be causing the XML data we're receiving from NOAA to not have a change observation time. It's possible that the system that generates the XML data (on the NOAA side) isn't as reliable as whatever system generates the metars report, but that's beyond my knowledge.

Here's an example of the XML that's returned for station KATT:

Code: Select all

<?xml version="1.0" encoding="ISO-8859-1"?> 
<?xml-stylesheet href="latest_ob.xsl" type="text/xsl"?>
<current_observation version="1.0"
	 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
	 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	 xsi:noNamespaceSchemaLocation="http://www.weather.gov/view/current_observation.xsd">
	<credit>NOAA's National Weather Service</credit>
	<credit_URL>https://weather.gov/</credit_URL>
	<image>
		<url>https://weather.gov/images/xml_logo.gif</url>
		<title>NOAA's National Weather Service</title>
		<link>https://www.weather.gov</link>
	</image>
	<suggested_pickup>15 minutes after the hour</suggested_pickup>
	<suggested_pickup_period>60</suggested_pickup_period>
	<location>Austin City, Austin Camp Mabry, TX</location>
	<station_id>KATT</station_id>
	<latitude>30.31667</latitude>
	<longitude>-97.76667</longitude>
	<observation_time>Last Updated on Jan 30 2022, 3:51 am CST</observation_time>
        <observation_time_rfc822>Sun, 30 Jan 2022 03:51:00 -0600</observation_time_rfc822>
	<weather>Fair</weather>
	<temperature_string>41.0 F (5.0 C)</temperature_string>
	<temp_f>41.0</temp_f>
	<temp_c>5.0</temp_c>
	<relative_humidity>57</relative_humidity>
	<wind_string>Calm</wind_string>
	<wind_dir>North</wind_dir>
	<wind_degrees>0</wind_degrees>
	<wind_mph>0.0</wind_mph>
	<wind_kt>0</wind_kt>
	<pressure_string>1022.5 mb</pressure_string>
	<pressure_mb>1022.5</pressure_mb>
	<pressure_in>30.20</pressure_in>
	<dewpoint_string>27.0 F (-2.8 C)</dewpoint_string>
	<dewpoint_f>27.0</dewpoint_f>
	<dewpoint_c>-2.8</dewpoint_c>
	<visibility_mi>10.00</visibility_mi>
 	<icon_url_base>https://forecast.weather.gov/images/wtf/small/</icon_url_base>
	<two_day_history_url>https://www.weather.gov/data/obhistory/KATT.html</two_day_history_url>
	<icon_url_name>nskc.png</icon_url_name>
	<ob_url>https://www.weather.gov/data/METAR/KATT.1.txt</ob_url>
	<disclaimer_url>https://www.weather.gov/disclaimer.html</disclaimer_url>
	<copyright_url>https://www.weather.gov/disclaimer.html</copyright_url>
	<privacy_policy_url>https://www.weather.gov/notice.html</privacy_policy_url>
</current_observation>

[jay (support)]

BTW, you'll note in my post above that the KATT station, the Austin airport, hasn't updated since early this morning.

In fact, all of the stations that I've set up for testing (all of yours and a couple for me) have been dark since at least the same time this morning if not earlier. I'm beginning to suspect that this NOAA data source is just no longer very reliable...

I'm going to keep an eye on this for the next month or so. If the data continues to be generally spotty (as opposed to spotty for just a few stations), we may have to stop including it with Indigo and put a big warning in the plugin store that the source doesn't appear reliable. If anyone has any contacts in NOAA, now would be a good time to see if they know anything.

[dtich]

Thanks for sharing all that Jay. And on a Sunday.

So, this seems like it might be a NWS issue. What I see are the XMLs in their DB are as the plugin indicates: old by 6 hours or so now. BUT, the current raw observation data on the site is up-to-date for the last hour.

It seems the stations are updating but NWS/NOAA is not making XMLs...? Really don't get that. Perhaps they're having a system-wide issue with file formatting or database handling.... I see no errors anywhere on their site, no red flags. This seems really dangerous to me. But I suppose only non-critical civilian users get the info from the XML? No idea really.

Hm. Head scratching. Wish there were someone at NWS who could explain.

Thanks for helping track this down.

[dtich]

@Jay, just saw your latest post. Yes, seems quite unreliable atm.

Another alternative is to access the METAR directly, the raw obs data, comes in a text string like the ones I posted. Relatively easy to parse.

This would be an example: https://w1.weather.gov/data/METAR/KVNY.1.txt

It doesn't contain as many data fields as the full XML, which is augmented by other DBs, but it is the latest from the source and contains all the basics. FWIW.

I can really only think that NWS is having an issue and it will be solved. How and why would they continue with XMLs being out of date and laggy? Seems like an internal issue that oughta be fixed asap.

Yes, if anyone has a connection at NOAA/NWS would be great to know the thinking on this.


Thanks all.

[FlyingDiver]

I'm thinking the XML feed is going away, and we should probably be using the JSON feed at https://www.weather.gov/documentation/services-web-api

[dtich]

Yes. I wonder if they're moving toward this only. Nice of them to tell us.

the GET /stations/{stationId}/observations/latest

function tester on that linked page does indeed return all the fields and more.

this:

Code: Select all

curl -X GET "https://api.weather.gov/stations/KVNY/observations/latest?require_qc=false" -H "accept: application/geo+json"

returns this:

{
"@context": [
"https://geojson.org/geojson-ld/geojson-context.jsonld",
{
"@version": "1.1",
"wx": "https://api.weather.gov/ontology#",
"s": "https://schema.org/",
"geo": "http://www.opengis.net/ont/geosparql#",
"unit": "http://codes.wmo.int/common/unit/",
"@vocab": "https://api.weather.gov/ontology#",
"geometry": {
"@id": "s:GeoCoordinates",
"@type": "geo:wktLiteral"
},
"city": "s:addressLocality",
"state": "s:addressRegion",
"distance": {
"@id": "s:Distance",
"@type": "s:QuantitativeValue"
},
"bearing": {
"@type": "s:QuantitativeValue"
},
"value": {
"@id": "s:value"
},
"unitCode": {
"@id": "s:unitCode",
"@type": "@id"
},
"forecastOffice": {
"@type": "@id"
},
"forecastGridData": {
"@type": "@id"
},
"publicZone": {
"@type": "@id"
},
"county": {
"@type": "@id"
}
}
],
"id": "https://api.weather.gov/stations/KVNY/observations/2022-01-30T16:51:00+00:00",
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-118.48,
34.22
]
},
"properties": {
"@id": "https://api.weather.gov/stations/KVNY/observations/2022-01-30T16:51:00+00:00",
"@type": "wx:ObservationStation",
"elevation": {
"unitCode": "wmoUnit:m",
"value": 244
},
"station": "https://api.weather.gov/stations/KVNY",
"timestamp": "2022-01-30T16:51:00+00:00",
"rawMessage": "KVNY 301651Z AUTO 27006KT 10SM CLR 14/M06 A3015 RMK AO2 SLP205 T01391061",
"textDescription": "Clear",
"icon": "https://api.weather.gov/icons/land/day/skc?size=medium",
"presentWeather": [],
"temperature": {
"unitCode": "wmoUnit:degC",
"value": 13.9,
"qualityControl": "V"
},
"dewpoint": {
"unitCode": "wmoUnit:degC",
"value": -6.1,
"qualityControl": "V"
},
"windDirection": {
"unitCode": "wmoUnit:degree_(angle)",
"value": 270,
"qualityControl": "V"
},
"windSpeed": {
"unitCode": "wmoUnit:km_h-1",
"value": 11.16,
"qualityControl": "V"
},
"windGust": {
"unitCode": "wmoUnit:km_h-1",
"value": null,
"qualityControl": "Z"
},
"barometricPressure": {
"unitCode": "wmoUnit:Pa",
"value": 102100,
"qualityControl": "V"
},
"seaLevelPressure": {
"unitCode": "wmoUnit:Pa",
"value": 102050,
"qualityControl": "V"
},
"visibility": {
"unitCode": "wmoUnit:m",
"value": 16090,
"qualityControl": "C"
},
"maxTemperatureLast24Hours": {
"unitCode": "wmoUnit:degC",
"value": null
},
"minTemperatureLast24Hours": {
"unitCode": "wmoUnit:degC",
"value": null
},
"precipitationLastHour": {
"unitCode": "wmoUnit:m",
"value": null,
"qualityControl": "Z"
},
"precipitationLast3Hours": {
"unitCode": "wmoUnit:m",
"value": null,
"qualityControl": "Z"
},
"precipitationLast6Hours": {
"unitCode": "wmoUnit:m",
"value": null,
"qualityControl": "Z"
},
"relativeHumidity": {
"unitCode": "wmoUnit:percent",
"value": 24.48191577848,
"qualityControl": "V"
},
"windChill": {
"unitCode": "wmoUnit:degC",
"value": null,
"qualityControl": "V"
},
"heatIndex": {
"unitCode": "wmoUnit:degC",
"value": null,
"qualityControl": "V"
},
"cloudLayers": [
{
"base": {
"unitCode": "wmoUnit:m",
"value": null
},
"amount": "CLR"
}
]
}
}

[jay (support)]

We'll see what we can do for the next release...

[dtich]

We'll see what we can do for the next release...

Da best. Thanks Jay.

[dtich]

I might be seeing a pattern of stations updating around 230A and 230P.... perhaps they have switched the XML creation schedule to every 12 hrs.... which is a real bummer. But hopefully we'll circumvent this unreliable bs they're into now. A little crazy that there is no announcement about this from NWS. Unless it's just regular old failing, which.. not sure which is better.

[dtich]

Update:

I managed to get an email exchange going with tech supp/webmaster at NWS, he's been very kind and this is the latest response from him:

Hello Dylan. Thanks for the link.

Are you using a script to pull the information, or is someone manually looking at this link each hour?

If you are using a script, would it be possible to add a random string after the url that changes each time the script polls the website? I like to use the unix timestamp because I know that will be unique every time. For example, something like this:

https://w1.weather.gov/xml/current_obs/ ... 1643778123

My theory is that it is a caching issue. A unique URL will trick whatever machine is polling the data into thinking that it is a new site that it has never gone to before.

Let me know.

Thanks

@Jay, what do you think? Does this sound plausible?

Anyway, FYI. Also, he may have broken some sort of logjam free on his end as the last day has seen pretty much consistent hourly updates I think. Better than a few days ago for sure. Haven't slipped to the first fallback station in 24 hours I don't think. FWIW.

Edit: This is the address in case you want to connect directly ever: [email protected]

[jay (support)]

@Jay, what do you think? Does this sound plausible?

No. We're not using a browser (which might be susceptible to caching). We're directly accessing the URL every time without any possibility of caching. What we get is what they're returning every time.

[dtich]

That's what I told him.

Ok thx.

[dtich]

NOAA stations reported very consistently for a couple weeks, then went awol again. i notified the webmaster and this is latest response. just fyi. not really sure what we do about this if the caching is on their end... just to update:

Hello Dylan. I apologize for the issues. Were you able to try the random string trick at the end of the url. As I mentioned before, it may not be so much an issue with caching done at the local level, but from our akamai caching service at the server level. This trick would help if that is the issue. Anothing think I would be curious about is the next time your script pulls old data, manually visit the page on your web browser and let me know if the data is still old, if so, try it again manually with a ?randomstring .

I am also sending your issue up to our regional and national web teams.

Here is a summary of the issue:

Dylan is using a script to grab xml feeds like: https://w1.weather.gov/xml/current_obs/KVNY.xml but the data is not being updated in a consistent manner. Often times the data returned is old. The data is being used to control irrigation systems. Any additional input would be appreciated.

Thanks.

and then this:

Hi Dylan,

Here at the regional headquarters, we typically use the APIs to gather our observation data. We often use synoptic, which will provide you with data in json format or xml (https://developers.synopticdata.com/mesonet/explorer/). Weather.gov also has an API for forecast data, which is very stable (https://www.weather.gov/documentation/services-web-api).

I'm not involved with the maintenance of the w1.weather.gov xml service, but just wanted to let you know what I use as a data source.


--
Gary Abadian
Western Region AWIPS/Web Program Manager
National Weather Service, Western Region Headquarters
Desk (385) 419-3135
Cell (801) 300-3456