NOAA Weather Plug-in

[weisswriter]

I'd like to see the NOAA Weather plug-in retrieve FORECAST information, so I could cancel watering on days when rain is forecast for the NEXT day or when rain has occurred on the current day. At the moment, I can do the latter but not the former. Ideally, it would retrieve the POP (probability of precipitation) and I could then decide based on the likelihood of rain whether or not to water. I do not currently see a way to do this with the NOAA or any other plug-in. If someone has figured it out, please tell me how.

[jay (support)]

NOAA doesn't provide forecast data - which is why it's not there.

[weisswriter]

It doesn't? When we say NOAA we really mean the National Weather Service (NOAA is the parent agency), which certainly does offer forecast information. If it didn't, 99.5% of the TV weather people wouldn't know what to say. I'm currently looking at the forecast for my area, through next Monday. Maybe you're saying it isn't in a form/format that lets it be accessed and used?

From the official NOAA website:

Covering the sun to the seas, the National Weather Service provides local and regional forecasts, and emergency alerts for severe storms, tornadoes, hurricanes, floods, extreme heat, winter storms, fire threats, tsunamis and solar flares. From its national centers to its 122 Weather Forecast Offices and 13 River Forecast Centers, the National Weather Service is watching over the nation and your neighborhood.

[jay (support)]

- ok, I'll be more specific. The NOAA data feed that we use doesn't provide forecast data. It's the only data feed that contains temp, humidity, and wind data which is the primary data we're looking to provide. Unfortunately it doesn't provide other data (like rain) which is why we built the plugin to WeatherSnoop.

I believe there was a feed I found that had forecast information but we decided to not to use it for several reasons:

• It would have been challenging to match the NOAA weather station ID we use for the first feed to the forecast data feed since the data comes from two completely different systems at NOAA. I'm sure that the TV weather guys get all this data and have some great stuff that aggregates it - but that costs $$.
• The data is a verbose block of text which has no practical uses in automation because it's not structured. This is further complicated by the fact that we don't have a good way to display blocks of text on Control Pages - something we want to address in a future version of Indigo but we don't support yet.

We didn't find any other feeds from NOAA (even after talking with them) which would have supplied that data in an easily parseable format and without contract (read $$).

So, while NOAA has the data, it's not easy to get to all of it (nor is all of it available for free). However, if you know of a free data feed that supplies that data then I certainly encourage you to build a plugin that provides that data - that's why we built the plugin API.

[weisswriter]

This and more info at http://graphical.weather.gov/xml/#XML_contents.

What is NDFD XML?

National Digital Forecast Database (NDFD) Extensible Markup Language (XML) is a service providing the public, government agencies, and commercial enterprises with data from the National Weather Service’s (NWS) digital forecast database. This service, which is defined in a Service Description Document, provides NWS customers and partners the ability to request NDFD data over the internet and receive the information back in an XML format. The request/response process is made possible by the NDFD XML Simple Object Access Protocol (SOAP) server. To see the details of the NDFD XML SOAP service, go to the following URL and click on the NDFDgen or NDFDgenByDay link:

http://graphical.weather.gov/xml/SOAP_s ... server.php

Why provide an NDFD XML service?

The National Weather Service is striving to serve society’s needs for weather information by evolving its services from a text-based paradigm to one based on making NWS information available quickly, efficiently, and in convenient and understandable forms. The NDFD is one example of this transformation. NDFD XML takes yet another step towards a digital services era by making NDFD data available for computer to computer transfer and processing. NWS customers and partners can then enhance the value of NDFD data through the creation of value added products.

----
I don't see anything about them charging for this. Since it's government information, we've all paid for it already and they seem anxious to share the wealth. It's structured. Looks like a straight ahead Web services call, but I'm not a programmer so I don't understand all the complexities (if I WERE a programmer, I'd do what you suggest and write a plug-in!). Or maybe this isn't a type of feed you can readily access using the tools available in Indigo. But it certainly looks as though the data is available - it's just a matter of getting and parsing it.

The list of accessible products for up to 7 days in advance (though you can control it and retrieve any number of days up to 7) is


Maximum Temperature maxt
Minimum Temperature mint
3 Hourly Temperature temp
Dewpoint Temperature dew
Apparent Temperature appt
12 Hour Probability of Precipitation pop12
Liquid Precipitation Amount qpf
Snowfall Amount snow
Ice Accumulation iceaccum
Cloud Cover Amount sky
Relative Humidity rh
Wind Speed wspd
Wind Direction wdir
Weather wx
Weather Icons icons
Wave Height waveh
Probabilistic Tropical Cyclone Wind Speed >34 Knots (Incremental) incw34
Probabilistic Tropical Cyclone Wind Speed >50 Knots (Incremental) incw50
Probabilistic Tropical Cyclone Wind Speed >64 Knots (Incremental) incw64
Probabilistic Tropical Cyclone Wind Speed >34 Knots (Cumulative) cumw34
Probabilistic Tropical Cyclone Wind Speed >50 Knots (Cumulative) cumw50
Probabilistic Tropical Cyclone Wind Speed >64 Knots (Cumulative) cumw64
Wind Gust wgust
Fire Weather from Wind and Relative Humidity critfireo
Fire Weather from Dry Thunderstorms dryfireo
Convective Hazard Outlook conhazo
Probability of Tornadoes ptornado
Probability of Hail phail
Probability of Damaging Thunderstorm Winds ptstmwinds
Probability of Extreme Tornadoes pxtornado
Probability of Extreme Hail pxhail
Probability of Extreme Thunderstorm Winds pxtstmwinds
Probability of Severe Thunderstorms ptotsvrtstm
Probability of Extreme Severe Thunderstorms pxtotsvrtstm
Probability of 8- To 14-Day Average Temperature Above Normal tmpabv14d
Probability of 8- To 14-Day Average Temperature Below Normal tmpblw14d
Probability of One-Month Average Temperature Above Normal tmpabv30d
Probability of One-Month Average Temperature Below Normal tmpblw30d
Probability of Three-Month Average Temperature Above Normal tmpabv90d
Probability of Three-Month Average Temperature Below Normal tmpblw90d
Probability of 8- To 14-Day Total Precipitation Above Median prcpabv14d
Probability of 8- To 14-Day Total Precipitation Below Median prcpblw14d
Probability of One-Month Total Precipitation Above Median prcpabv30d
Probability of One-Month Total Precipitation Below Median prcpblw30d
Probability of Three-Month Total Precipitation Above Median prcpabv90d
Probability of Three-Month Total Precipitation Below Median prcpblw90d
Real-time Mesoscale Analysis Precipitation precipa_r
Real-time Mesoscale Analysis GOES Effective Cloud Amount sky_r
Real-time Mesoscale Analysis Dewpoint Temperature td_r
Real-time Mesoscale Analysis Temperature temp_r
Real-time Mesoscale Analysis Wind Direction wdir_r
Real-time Mesoscale Analysis Wind Speed wspd_r
Watches, Warnings, and Advisories wwa


And they're adding min/max relative humidity on 6/19!!

[lalisingh]

This and more info at http://graphical.weather.gov/xml/#XML_contents.

Tried it out with the simple applescript below and it works!! Thanks!

To use do:
1) Define and set indigo variable to WSforecasturl to http://forecast.weather.gov/MapClick.ph ... TextType=1
Do change the lat and lon though.
2) Create a schedule with the attached applescript.
3) View the result in the Indigo event log

Should be straighforward to clean up the html markup.

Code: Select all

local oldDelims, qChar
local target_URL
local noaaForecast

set oldDelims to AppleScript's text item delimiters -- save their current state
set AppleScript's text item delimiters to {","} -- declare new delimiters	
set qChar to ASCII character 34

-- get forecast from NOAA
-- define and set indigo variable to WSforecasturl to http://forecast.weather.gov/MapClick.php?lat=35.289000&lon=-106.468000&TextType=1
--
set the target_URL to value of variable "WSforecasturl"
log target_URL
set noaaForecast to "undefined"
try
	set noaaForecast to do shell script "curl -s --connect-timeout 2 --max-time 5 --get " & qChar & target_URL & qChar
end try
log noaaForecast

set AppleScript's text item delimiters to oldDelims -- restore them

[nsheldon]

I prefer the 12-hour chance of rain forecast provided by Weather.com. I'm not sure if it'll provide enough forecast for you, weisswriter, but I use it to have Indigo verbally tell me if there's likely to be rain within a few hours as I'm leaving the house. I wrote a little bash shell script to suck in the HTML output from Weather.com for my location, filter out all but the time and chance of precipitation for that time, and output the values as basic comma-separated-values text to Standard Out (so it can be further processed by, for instance, an AppleScript that populates values into Indigo). If you're interested, the shell script code is below.

Code: Select all

#!/bin/bash

# Weather Precipitation Parser
#
# by Nathan Sheldon ([email protected]) - 16-Mar-2012


# -- VARIABLES --
export file=/tmp/weather.html
export URL=http://www.weather.com/weather/hourbyhour/graph/Fresno+CA+93710:4:US
export hourlyPercent
export hour


# -- FUNCTIONS --
function getHourlyWeather ()
{
	curl -s $URL >$file  # -s = silent mode.

	return 0
}

function removeTempFiles ()
{
	rm $file
}


# -- MAIN --
getHourlyWeather

lineCount=0
searchOn=FALSE

while read line
do
	if [[ "$line" == *\"wx-time\"\>* ]]
	then
		# Remove HTML...
		hour[$lineCount]=${line/*wx-time\"\>/}
		hour[$lineCount]=${hour[$lineCount]/\<*/}
		# Add zero minutes (:00) to on-hour predictions...
		if [[ ${hour[$lineCount]} != *:* ]]
		then
			meridian=${line/*wx-meridian\"\>/}
			meridian=${meridian/\<*/}
			hour[$lineCount]=${hour[$lineCount]}$meridian
			if [[ ${hour[$lineCount]} == *AM ]]
			then
				hour[$lineCount]="${hour[$lineCount]/ AM/:00 AM}"
			elif [[ ${hour[$lineCount]} == *PM ]]
			then
				hour[$lineCount]="${hour[$lineCount]/ PM/:00 PM}"
			fi
		else
			hour[$lineCount]=${hour[$lineCount]}$meridian
		fi
		# Convert 12-hour time to 24-hour time...
		if [[ ${hour[$lineCount]} == *AM ]]
		then
			hour[$lineCount]="${hour[$lineCount]/ AM/}"
			if [[ ${hour[$lineCount]/:*/} == 12 ]]
			then
				hour[$lineCount]="${hour[$lineCount]/12:/0:}"
			fi
		else
			hour[$lineCount]="${hour[$lineCount]/ PM/}"
			if ((${hour[$lineCount]/:*/} < 12))
			then
				tempHour="${hour[$lineCount]/:*/}"
				((tempHour=tempHour+12))
				hour[$lineCount]="${hour[$lineCount]/*:/$tempHour:}"
			fi
		fi

		# Convert to seconds since midnight...
		seconds1="${hour[$lineCount]/:*/}"
		seconds2="${hour[$lineCount]/*:/}"
		((seconds1=seconds1*3600))
		((seconds2=seconds2*60))
		((hour[$lineCount]=seconds1+seconds2))

		# Display result as first field of a CSV data line...
		echo -n "${hour[$lineCount]},"
	fi

	if [[ "$line" == *PRECIP* ]]
	then
		searchOn=TRUE
	fi

	if [[ $searchOn == TRUE ]]
	then
		if [[ "$line" == *\&\#37* ]]
		then
			# Get the percent chance of precipitation...
			hourlyPercent[$lineCount]="$line"
			# Remove HTML and percent sign...
			hourlyPercent[$lineCount]="${hourlyPercent[$lineCount]/*	/}"
			hourlyPercent[$lineCount]="${hourlyPercent[$lineCount]/\&\#37*/}"
			# Display result as second field of CSV data line.
			echo "${hourlyPercent[$lineCount]}"
			((lineCount++))
			searchOn=FALSE
		fi
	fi
done <$file

removeTempFiles

exit 0

Things to consider: You'll need to modify the "URL" variable in the above script for your location. Also, note that the output of the time value is in seconds from midnight (just like how AppleScript calculates time when you get the "time of (current date)" value). Time values lower than the current time, represent seconds from midnight of the next day (e.g. 0 would be midnight of tomorrow, not today). If you'd like the AppleScript I use to parse the time and percent chance of precipitation from the shell script output and populate Indigo variables with that info, let me know and I can post that as well.

[weisswriter]

NSheldon, thanks for the info. I WOULD appreciate seeing the AppleScript. Maybe I can glue it together and do a bit of self-education at the same time. I've done some scripting in various database and other situations over the years, but never got around to AppleScript.

The NWS forecast database, interestingly enough, also provides only a 12-hour probability-of-precip forecast. It's kind of odd, since NWS obviously forecasts the chance of rain a day or more in advance.

[nsheldon]

Yea, the NWS probability is probably the original source for the Weather.com data, however, the NWS data is in a more "human readable" format, which is actually a bit harder to parse than the format used by Weather.com. As for the limit of only 12 hours, I'm sure it has to do with the reliability of the prediction decreasing as the time of the prediction increases. After 12 hours in the future, it's probably not possible to reliably predict precipitation to within 1-hour increments. You'll notice that the Weather.com output actually has predictions for every 15 minutes for the hours closest to the current time. But for hours farther out, only 1 hour probabilities are given.

In any case, here's the AppleScript. I put both the shell script and the AppleScript below (saved in separate files) inside the /Library/Application Support/Perceptive Automation/Indigo 5/Scripts/Background Tasks folder. In order for the AppleScript below to work, both files must be in that location, and the shell script must be named "get_12hr_chance_of_precipitation.sh" (unless you change the file name in the AppleScript to something different). The below script will create the weather variables in Indigo if they don't exist, so there's no Indigo changes that you need to do, other than, perhaps, creating a Scheduled task to run the below script (as an external script file, not an embedded one) on a regular basis. In my case, I have Indigo run this script every 15 minutes.

Code: Select all

(*
by Nathan Sheldon

Version: 1.0

Revisions:	1.0 (18-Mar-2012)
			Initial release

This Indigo Pro attachment executes the custom shell script "get_12hr_chance_of_precipitation.sh" in the Background Tasks folder, assigning the output from that script to variables in Indigo.  The output from that shell script is a basic CSV with the predicted chance of precipitation for the next 12 hours from Weather.com for the 93710 zip code.  There are 2 fields per line.  The first field is the number of seconds since midnight.  The second field is the percentage chance of precipitation for that time period.  There's one time period per line.

NOTE: Predictions for the next 12 hours generated after 12:00 PM will include time periods after midnight of the following day, but will have time representations less than the current time of day and should be assumed to be predictions for the following day, not predictions for previous time periods of the current day.

This script will automatically create any Indigo weather variable that doesn't already exist.
*)

-- Get the percent chance of precipitation for the next 12 hours using the custom "get_12hr_chance_of_precipitation.sh" shell script.

set thePredictions to ""
set theTime to 0
set thePercent to 0
set highestPercent to 0
set highestPercentTime to 0

-- First erase the values in all the Weather_Chance_of_Precip_* variables.
tell application "IndigoServer"
	set theVariables to the name of every variable whose name starts with "Weather_Chance_of_Precip_"
	repeat with thisVariable in theVariables
		set value of variable thisVariable to ""
	end repeat
end tell

set thePredictions to do shell script "/Library/Application\\ Support/Perceptive\\ Automation/Indigo\\ 5/Scripts/Background\\ Tasks/get_12hr_chance_of_precipitation.sh"

tell application "IndigoServer"
	repeat with x from 1 to (count of paragraphs in thePredictions)
		set theData to paragraph x of thePredictions
		set AppleScript's text item delimiters to ","
		set theTime to text item 1 of theData
		set thePercent to text item 2 of theData
		set AppleScript's text item delimiters to {""}
		if thePercent > highestPercent then
			set highestPercent to thePercent
			set highestPercentTime to theTime
		end if
		if exists variable "Weather_Chance_of_Precip_Highest" then
			set value of variable "Weather_Chance_of_Precip_Highest" to highestPercentTime & "," & highestPercent
		else
			make new variable with properties {name:"Weather_Chance_of_Precip_Highest", value:highestPercentTime & "," & highestPercent}
		end if
		if exists variable ("Weather_Chance_of_Precip_" & x) then
			set value of variable ("Weather_Chance_of_Precip_" & x) to theTime & "," & thePercent
		else
			make new variable with properties {name:"Weather_Chance_of_Precip_" & x, value:theTime & "," & thePercent}
		end if
	end repeat
end tell

[weisswriter]

Thanks for that, NSheldon. I appreciate it.

While I'm sure accuracy of the precip forecast drops as you go farther out, that's true of ALL forecasts. And NWS certainly issues a "chance of rain on Tuesday" sort of forecast several days ahead, so you'd think they must calculate the probabilities farther out. But maybe not a formal numerical calculation.

[nsheldon]

Thanks for that, NSheldon. I appreciate it.

Glad I could help. Hope it's useful.

[jay (support)]

Another plugin idea for some enterprising developer...

[nsheldon]

Another plugin idea for some enterprising developer...

LOL! Indeed! Actually, that's a good idea. This would probably be about the easiest plugin to start with. I'll look into doing that I think.

[weisswriter]

With the kind assistance of Eric Boldt, the Los Angeles/Oxnard NWS Warning Coordination Meteorologist (whom I e-mailed), I found that the NWS forecast database I cited yesterday in fact has precipitation probabilities in 12 hour chunks to 168 hours (7 days) out! The full and technically correct list of what can be retrieved from the Weather Service's XML database can be found at http://www.nws.noaa.gov/ndfd/technical.htm#elements.

One of the data elements is "quantitative precipitation amount," so they are forecasting not only the CHANCE it will rain, but the expected AMOUNT for up to 72 hours out. That could be useful.

So with THAT in hand, I hope NSheldon or some other enterprising and capable dev will be inspired to create a plug-in that pulls probability of precipitation data (along with whatever else seems useful) directly from the horse's mouth for at least a day or two (or a user-selectable period) in advance, and get it into Indigo where it can be used to make irrigation (and perhaps other) decisions.

[nsheldon]

The full and technically correct list of what can be retrieved from the Weather Service's XML database can be found at http://www.nws.noaa.gov/ndfd/technical.htm#elements.

Ha... "technically correct". Indeed. I checked that out. Yes. It's technically correct in that the data exists in the database. However, the NWS seems to be frustratingly cryptic about exactly HOW to access that data. After a few hours of reading their deceptively specific (but imprecise) content and following links all over the place, it turns out that it's actually rather simple to request specific data (though the data is referred to by different names on different pages, making things decidedly confusing).

In short, you can fill out the form on this page and it returns the XML containing the data you requested. Perhaps even more convenient (at least for scripting purposes), the form elements are submitted in the URL itself (as opposed to being sent in an HTTP POST request). Note, however, that the "quantitative precipitation amount" you referred to appears on that page as "QPF" (Quantitative Precipitation Fluid), while in the generated XML, that data is referred to as "Liquid Precipitation Amount". Yay for consistency.

...so they are forecasting not only the CHANCE it will rain, but the expected AMOUNT for up to 72 hours out. That could be useful.

Mmmm... I wouldn't really consider the amount of rain predicted to fall in a time period to be quite the same as a measure of the probability of the existence of precipitation (for example, it could rain 0.1 inches in an hour with a PoP of 80% or it could rain 1.5 inches in an hour with the same PoP of 80%). However, I'm sure that wasn't your point. And I agree. The predicted amount of precipitation could be useful. I'd even go further and say it's MORE useful than a simple percent chance.

So with THAT in hand, I hope NSheldon or some other enterprising and capable dev will be inspired to create a plug-in that pulls probability of precipitation data (along with whatever else seems useful) directly from the horse's mouth for at least a day or two (or a user-selectable period) in advance, and get it into Indigo where it can be used to make irrigation (and perhaps other) decisions.

If one were to consider me an "enterprising and capable dev", it sounds to me as though you'd fall into that category as well