Data Issues

Posted on
Tue May 15, 2018 4:09 am
DaveL17 offline
User avatar
Posts: 5349
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Data Issues

Weather Underground is having some issues with its API--it is occasionally returning bad data for forecasts (there might be issues in other areas, too). One user saw bad high/low data and bad dates for 10-day forecast devices. At first I thought that this was only a condition when the language preference was set to Dutch, but now I'm not so sure. Today, I woke up to a forecast high temperature for tomorrow of -17F and a low of -6. I then asked for a weather data refresh from the plugin menu and the data were accurate again. This was with a language preference of English.

Note that the plugin does no calculations on these data and the language preference is handled on the WU side--I simply post the data provided by the API.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue May 15, 2018 5:41 am
DaveL17 offline
User avatar
Posts: 5349
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Data Issues

Several reports of the data issue on the Weather Underground forums. I've sent a message to their tech support (although others have also).

https://apicommunity.wunderground.com/weatherapi

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue May 15, 2018 8:06 pm
DaveL17 offline
User avatar
Posts: 5349
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Data Issues

Thanks for the suggestion. The plugin already caches on the API age (for the entire package delivery)--if the delivered data are older than data we already have, the plugin won't update--but I think that caching on a delta value or components of the package might be pretty tough. To think "out loud":

What would be a good delta for temperature? I think there might actually need to be two--one for ºF and one for ºC. I don't think it's a delta percent because that wouldn't apply in the same way across the temperature range (10% of 80ºF is 8º whereas 10% of 20ºF is 2º).

Controlling for refresh cycle time might also be pretty complicated. I set my plugin to update every 15 minutes, but someone else might update only once or twice a day. With a longer refresh window, larger swings in data would be normal.

I need to think about this some more. What you suggest makes perfect sense, but I don't think it would be trivial and I'd have to think about it across a vast array of values (temp, precip, pressure, wind, etc.) I've done some preliminary research on how to handle outliers with the matplotlib plugin and the answer is much easier to conceptualize than to implement. It's not that it's not possible....it just ain't particularly easy. :D

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat May 19, 2018 9:29 pm
RogueProeliator offline
User avatar
Posts: 2309
Joined: Nov 13, 2012
Location: Baton Rouge, LA

Re: Data Issues

I personally wouldn't do anything to try to compensate as there will ALWAYS be something or someone that comes up with some situation that doesn't work for your solution. I think a better approach is to allow the users to make error corrections when needed. For example, just displaying on my weather page I don't care if it has a bad value; but in a trigger to change the air handling I might want to but some reality check in place and that can be done by the user in an action/trigger/script.

Just my $0.02

Posted on
Sun May 20, 2018 5:07 am
DaveL17 offline
User avatar
Posts: 5349
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Data Issues

RogueProeliator wrote:
I personally wouldn't do anything to try to compensate as there will ALWAYS be something or someone that comes up with some situation that doesn't work for your solution. I think a better approach is to allow the users to make error corrections when needed. For example, just displaying on my weather page I don't care if it has a bad value; but in a trigger to change the air handling I might want to but some reality check in place and that can be done by the user in an action/trigger/script.

Just my $0.02

Thanks for the thought. That's where I'm landing too. There's too many variables/scenarios to control for with a universal solution. I think the right course is for the user to build safety nets into their automation where it's mission critical.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun May 27, 2018 9:49 pm
Professor Falken offline
User avatar
Posts: 299
Joined: Mar 29, 2015

Re: Data Issues

So I guess we're just kinda stuck if Wunderground is pushing out screwy data? I'm getting -99 inches of rain a day, which needless to say, has destroyed my sprinkler control scripts.

Posted on
Mon May 28, 2018 4:06 am
DaveL17 offline
User avatar
Posts: 5349
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Data Issues

You should always ignore a value of -99 -- it's a signal from WU (or from the plugin) that they don't know what a valid value is.

For your sprinklers, you could create a variable to store the precipitation value, update the variable if the plugin delivers a larger number, and reset the variable at some regular interval. Then, key your sprinkler off the variable value instead of the device state.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue May 29, 2018 10:02 am
Professor Falken offline
User avatar
Posts: 299
Joined: Mar 29, 2015

Re: Data Issues

Yeah, I'm going to have to watch what gets reported carefully, or maybe set up some logging or something I can review. It seems like sometimes the daily rainfall is reporting, while most of the time it isn't. This gets particularly messy since in my current setup I have a script set up to move the measured rainfall from "today" to "yesterday" and "yesterday" to the "day before yesterday" at midnight. If those numbers are screwy when that script runs, all the totals get all messed up without an easy way to fix them.

I also think the hourly rain reporting isn't working right either. If it was, I could probably just total the hourlies every hour to get a useful running daily.

Weird that all this was working so well last year, and has completely gone to pot this year. Maybe it's time to install my own station and just monitor it locally with weathersnoop. My problem is, I don't think I have a good place on my property that works well for both a rain gauge and an anemometer.

Or I could revert to letting the Rachio handle the weather skips. I abandoned their system, though, because it gave too much credence to future predicted rain that didn't end up happening.

Thanks for the suggestions.

Posted on
Tue May 29, 2018 7:56 pm
DaveL17 offline
User avatar
Posts: 5349
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Data Issues

Hey, you're welcome. WU have been having trouble with their API of late. There are many messages on their forums with similar reports to what you've been experiencing.

I would recommend that anyone tying their automation to WU's data definitely include a safety net.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 1 guest