Curl Form Post from Script (AS or Python) - Any Ideas?

Hey all,

I have done this in both python and AppleScript. I'm just looking for additional ideas from someone who might be a better programmer than I am.

The short story: I have an HTD.COM multi-zone audio system. Last week they finally shipped the pre-orders of the HTD network linc that lets me hit a web page on a small serial/IP device that plugs into the amp to get a remote control interface to the audio system zones/sources, etc.

I thought it would be a rest API, but instead is a weird .HTML page (ip.ip.ip.ip/keypad.html) with a form in it. The remote UIis in a table and just has a bunch of form fields with an onrelease submit.form action. It posts back to itself.

Anyway, I want to simulate these form field clicks by using cURL from indigo. The idea is to turn audio zones on or off, adjust volume, etc along with activities in Indigo.

I've done this first in Python: (and then Apple Script):

Code: Select all

import urllib
import urllib2

url = 'http://www.example.com'
values = {'var' : 500}

data = urllib.urlencode(values)
response = urllib2.urlopen(url, data)
page = response.read()

When I execute this, I get an error in Indigo that reads:
<urlopen error (61, 'Connection refused')>

The Indigo logs show:

Action Group turn audio to sonos copy
Script Error embedded script: <urlopen error (61, 'Connection refused')>
Script Error Exception Traceback (most recent call shown last):

embedded script, line 8, at top level
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 124, in urlopen
return _opener.open(url, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 381, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 399, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 360, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 1118, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 1093, in do_open
raise URLError(err)
URLError: <urlopen error (61, 'Connection refused')>

I am now wondering if I have to send the headers or something as well.

I can access this page from my browser just fine.

From Applescript, I doing this:

Code: Select all

do shell script "curl -d http://192.168.15.130/keypad.html >" & " SrcDropDown=2&VolDropDown=25&_isgrp=0&_grpmsk=32&+grpsrc=0&_zsrc=11&_cznum=5"


That compiles and runs fine, but nothing seems to happen on the audio panel.
I am sure those form values are exactly the same as what gets sent when the form is submitted for the action.
In this case, I am changing the audio source. It doesn't seem to work, but I get no errors in Indigo.

Thoughts?


Thanks again!

I did some command line testing and the curl command does in fact alter the audio zones.... So I know it works, but am not sure how to fire it from Indigo.

For example, from a command prompt, I can execute this command and it works:

Code: Select all

curl -d 'VolDropDown=20&_isgrp=0&_grpmsk=32&+grpsrc=0&_zsrc=10&_cznum=5' http://192.168.15.130/keypad.html


This is in Applescript, and more complex than your example, but should get you started. I use this to update my weather data to PWS. You'll have to watch out for quotes. do shell script issues a terminal command, and as you notice the command is contained in quotes. If your command has quotes within it, you'll need to make sure it all parses out correctly.

Code: Select all

set urlStart to "http://www.pwsweather.com/pwsupdate/pwsupdate.php"
   
   tell application "System Events"
      set pwsResponse to do shell script "curl -G -d ID=XXXXXXX -d PASSWORD=XXXXXXX -d action=updateraw -d softwaretype=personal -d dateutc=" & utcDate & " -d baromin=" & baromin & " -d humidity=" & humidity & " -d tempf=" & tempf & " -d dewptf=" & dewptf & " -d winddir=" & winddir & " -d windspeedmph=" & windspeedmph & " -d windgustmph=" & windgustmph & " " & urlStart
   end tell

That has a lot of variable substitutions in it, which you don't appear to have.

Here's another example of a do shell script command that had quotes, that I had to escape with the \ character:

Code: Select all

-- routine to convert unix timestamp to Applescript time
on epoch2datetime(epochseconds)
   set myshell1 to "date -r "
   set myshell2 to " \"+%m/%d/%Y %I:%M %p\""
   set theDateTime to do shell script (myshell1 & epochseconds & myshell2)
   return theDateTime
end epoch2datetime


I don't do curl often enough to know all the ins and outs. You may not need the single quotes in your example, and you could also break each element up with '-d' flags, as I did, if necessary.

Greg

Thanks so much for this.

I actually got it to work last night and realized that the way that the web site works is by relying on a session, so I have to send a couple of requests to it.

The first request:

Code: Select all

do shell script "/usr/bin/curl -d 'SrcDropDown=6&VolDropDown=15&BtnZ05=Basement&_isgrp=0&_grpmsk=32&+grpsrc=0&_zsrc=11&_cznum=5' http://192.168.15.130/keypad.html "

This selects the button for the specific zone (in this case, the basement)

The second request:

Code: Select all

do shell script "/usr/bin/curl -d 'SrcDropDown=6&VolDropDown=25&_isgrp=0&_grpmsk=32&+grpsrc=0&_zsrc=11&_cznum=5' http://192.168.15.130/keypad.html "

This then excludes the zone I want to control, and instead, updates the volume for the zone selected in the first request.


I am still playing around with the nuances of the commands, but doing this does execute the command via CURL and triggers the panel in the room (basement) to change the volume.

I am still playing with the other variables to understand what priority they have when received by the form post.


Dave

I have a suggestion that may make your script a bit easier to maintain and modify in the future. While do shell script does what you need, it also has lots of peculiarities with quoting special characters which can make the process more difficult than it needs to be. To overcome that, you can place all of your shell commands into a shell script file. You can use TextEdit to create the file, but be sure to save it as plain text. You can place that script in the Indigo scripting folder, or better, create a bin folder in your HOME directory for your shell scripts.

Then just create a script with whatever commands you need in it. For example to do your task it might look like:

Code: Select all

#!/bin/sh
/usr/bin/curl -d 'SrcDropDown=6&VolDropDown=15&BtnZ05=Basement&_isgrp=0&_grpmsk=32&+grpsrc=0&_zsrc=11&_cznum=5' http://192.168.15.130/keypad.html
/usr/bin/curl -d 'SrcDropDown=6&VolDropDown=25&_isgrp=0&_grpmsk=32&+grpsrc=0&_zsrc=11&_cznum=5' http://192.168.15.130/keypad.html

exit 0

Assuming you saved this script as voldn.sh, you would then make it executable by entering these commands from the terminal:
cd ~/bin
chmod 755 voldn.sh

Then, modify your Indigo AppleScript to:

Code: Select all

do shell script "/Users/myaccount/bin/voldn.sh >/dev/null 2>&1 &"

Some explanations: The #!/bin/sh (which must be the first line) tells OS X to use the Bourne shell (/bin/sh) to interpret the commands in the file. exit 0 is not really required, but is a nice way to assure that the script exits cleanly and signals that it ran successfully. >/dev/null 2>&1 & tells OS X to discard any output from the command, have it run in the background and exit immediately. This is not required, but using it lets you use the command in an embedded script since it will not block Indigo.

You can also get much fancier, like adding options to the script so you could name it volctl.sh and call it with options like:
volctl.sh -v up to raise the volume
and
volctl.sh -v dn to lower the volume
etc., etc.

This is great. Thanks!

So, now that you guys opened the door, riddle me this:

What would be your approach to use the shell scripts saved out, but also passing Indigo variables to them?

Thanks so much for the info.

Dave

...What would be your approach to use the shell scripts saved out, but also passing Indigo variables to them? ...

Well, short of a course on writing shell scripts...

You can pass arguments to a shell script very easily. For example, given a script foo.sh:

Code: Select all

#!/bin/sh
echo $1

Entering ./foo.sh hello
will print
hello

Any arguments to a shell script (called positional parameters) are available to the script is special variables $1, $2, $3, etc.
So, if your AppleScript looked like:

Code: Select all

set theMessage to "hello"
do shell script "/Users/myhome/bin/foo.sh " & theMessage

it would return hello

However (there is always a caveat) to make good use of this argument passing you should probably learn some basic shell logic like:

Code: Select all

#!/bin/sh
if [ $1 == "up" ];then
    curl.... curl args...
fi

There are a number of tutorials on shell scripting. A quick google found this site.

I like to use the following code. It need to import pycurl, how do I get that one?

Thx

Karl
have done it in applescript, now trying to do the same in py .. just to learn and get going with py

Code: Select all

import pycurl
def cameraset():
   postData =  '{"ReplySuccessPage" : "image.htm", "ReplyErrorPage" : "errrimg.htm"  , "BrightnessControl" : "64", "ContrastControl" : "64", "SaturationControl" : "64", "AntiFlickerEnable" : "1", "Mirror" : "0", "ConfigSystemStream" : "Save"}'
   c = pycurl.Curl()
   c.setopt(pycurl.URL, 'https://192.168.1.71/setSystemStream')
   c.setopt(pycurl.POST, 1)
   c.setopt(pycurl.POSTFIELDS, postData)
   c.setopt(pycurl.USERPWD, 'uuuu:ppppp')
   c.perform()

cameraset()


[jay (support)]

Well, you'll need to install pycurl. I don't know where exactly you'd get it, but usually Python modules are delivered either as .egg files or with a setup.py file that you run to install it.

Note, however, that Python has httplib built-in for handling http connections so you don't really need pycurl ( though it may do some abstraction that makes it a bit easier).

On my learning voyage, I have tried to solve the same problem in different ways and I thought, I share my result for those who are on a similar journey... ... from screen scraping to apple script to py ...

1. problem: the Dlink 930 is a very cheap camera, but does not adopt to changing brightness well. It has a web page where the sensitivity can be changed.

2. three solutions - all of them work.

(A) using safari and automating commands in applescript change the parameters. see code (A)
- pros if other tools not available ...
- cons, not stable, sensitive to changes, opens and closes safari, must be run in foreground and indigo waits until its finished, that creates timeout issues.
use as embedded script or as file

(B) applescript calling curl, launched as .app in background and curl is used with '&' to make it a background job. see code (B)
- pros: works well, easier to understand, good start for beginner.
- cons: have to understand the curl command .. used wireshark(in linux Vbox on MAC) fiddler(in vbox Windows7) .. to understand the communication
code should be in the ..../indigo 6 /scripts/attachment directory
In applescript editor do export and save as app, then do plugins/ reload attachments in indigo

(C) Phyton solution using urllib. see code (C)
- pros: goes with indigo philosophy... good translation project to go from applescript to PY
- cons:a little bit steeper learning curve
- launch py from within py and use & to send to background
(could also be call from applescript in embedded script with:
do shell script " '/Library/Application Support/Perceptive Automation/Indigo 6/Scripts/Attachments/setCameraLevels.py' 1 1 3 1 0 '192.168.1.71' >/dev/null 2>&1 & )

code should be in the ...../indigo 6 /scripts/attachment directory as referenced in the call statement.


comments welcome, especially for the PY section..
Karl

ps userids and passwords have been 'xxx'ed

Code: Select all

(*
----- CODE (A) ----
set brightness level of Dlink 930 camera through safari session
get ip number and brightness level from Indigo.
keep userid and password in code, does not change between cameras.

Karl Wachs
v 1.0
10/1/13


*)


set theUserID to "xxx"
set thePassword to "xxx"



try
   tell application "IndigoServer"
      set ipNumber to (get value of variable "cameraBrightnessLevelPNumber")
      set brightnessLevel to (get value of variable "cameraBrightnessLevel")
   end tell
on error
   return
end try



-- first login if needed
tell application "Safari"
   activate
   delay 0.5
   make new document
   set URL of document 1 to "http://" & ipNumber & "/image.htm"
   
end tell


set returnValue to SafariWindowIsLoading("Password")


if returnValue = "false" or returnValue = "true" then -- can we skip userid and password page?
   delay 4
   tell application "System Events"
      tell process "Safari"
         delay 0.1
         keystroke theUserID
         key code 48
         keystroke thePassword
         delay 0.1
         keystroke return
         delay 0.1
      end tell
   end tell
   -- close window
   tell application "Safari" to close every window
   delay 0.1
end if



--now start without /after login:
if returnValue is not equal to "skip" then -- is safari already on right page? if yes skip
   tell application "Safari"
      activate
      make new document
      set URL of document 1 to "http://" & ipNumber & "/image.htm"
   end tell
   SafariWindowIsLoading("Brightness") -- wait for "brighness" on the page
   delay 4
   
end if

tell application "System Events"
   tell process "Safari"
      -- tab to brightneslevel set field
      key code 48
      key code 48
      key code 48
      key code 48
      key code 48
      key code 48
      
      -- set brightness to top level no matter what, move up 10 times
      set ii to 0
      repeat until ii = 10
         delay 0.1
         key code 126
         set ii to ii + 1
      end repeat
      
      
      -- we are now at brightness +5,  decrement to right value
      set ii to 0
      repeat until ii = 5 - brightnessLevel
         delay 0.1
         key code 125
         set ii to ii + 1
      end repeat
      
      -- now save settings
      delay 0.1
      keystroke return
      
      -- tabe to save button
      key code 48
      key code 48
      key code 48
      keystroke return -- hit save button
      delay 3
      
   end tell
end tell

tell application "Safari" to close every window



tell application "IndigoServer"
   set value of variable "cameraBrightnessLevelRunning" to "false"
end tell



-----------------

on SafariWindowIsLoading(theText)
   
   delay 0.2
   
   
   repeat with i from 1 to 15
      tell application "Safari"
         if name of current tab of window 1 is not "Loading" then exit repeat
      end tell
      delay 1
   end repeat
   if i is 15 then return false
   tell application "Safari"
      repeat until (do JavaScript "document.readyState" in document 1) is "complete"
         delay 0.5
      end repeat
   end tell
   
   tell application "Safari" to set PageText to the text of document 1
   
   
   if PageText contains "Brightness" then return "skip"
   if PageText contains "theText" then return "true"
   return "false"
   
end SafariWindowIsLoading



Code: Select all

(* 

----- CODE (B) ------
set level parameters in DLINK 930 camera

-- example call in indigo scheduled action in embedded applescript
setCameraLevels(0, 0, 0, 1, 0, "192.168.1.75")
setCameraLevels(0, 0, 0, 1, 0, "192.168.1.77")
setCameraLevels(0, 0, 0, 1, 0, "192.168.1.78")

Karl Wachs
v1
10/1/13

*)

on setCameraLevels(brightnessLevel525, contrastControl525, SaturationControl525, AntiFlickerEnable01, mirrorOnOff01, theIPNumber)
   
   set theUserID to "xxx"
   set thePassword to "xxxx"
   --    map input(-5 ..+5)          -5--> 1 , .. , +5 --> 128     levels on webpage
   set mapping to {1, 14, 26, 39, 51, 64, 77, 90, 103, 115, 128}
   
   if brightnessLevel525 > 5 then brightnessLevel525 = 5
   if brightnessLevel525 < -5 then brightnessLevel525 = -5
   set setbrightnessLevel to item (brightnessLevel525 + 6) of mapping
   
   
   if contrastControl525 > 5 then contrastControl525 = 5
   if contrastControl525 < -5 then contrastControl525 = -5
   set setcontrastControl to item (contrastControl525 + 6) of mapping
   
   if SaturationControl525 > 5 then SaturationControl525 = 5
   if SaturationControl525 < -5 then SaturationControl525 = -5
   set setSaturationControl to item (SaturationControl525 + 6) of mapping
   
   if AntiFlickerEnable01 > 0 then set setAntiFlickerEnable to "1"
   if AntiFlickerEnable01 < 1 then set setAntiFlickerEnable to "0"
   
   if mirrorOnOff01 > 0 then set setMirror to "1"
   if mirrorOnOff01 < 1 then set setMirror to "0"
   
   
   -- compile cmd string   
   set cmd to " curl -u " & theUserID & ":" & thePassword -- userid and password
   --   set cmd to cmd & " -m 20 " -- max wait 20 secs
   
   --                           data
   set cmd to cmd & " -d  \"" & "ReplySuccessPage=image.htm&ReplyErrorPage=errrimg.htm"
   set cmd to cmd & "&BrightnessControl=" & setbrightnessLevel
   set cmd to cmd & "&ContrastControl=" & setcontrastControl
   set cmd to cmd & "&SaturationControl=" & setSaturationControl
   set cmd to cmd & "&AntiFlickerEnable=" & setAntiFlickerEnable
   set cmd to cmd & "&Mirror=" & setMirror
   set cmd to cmd & "&ConfigSystemStream=Save\"" -- save it
   --                            end data    
   
   set cmd to cmd & " http://" & theIPNumber & "/setSystemStream" -- this is the page we are addressing
   
   set cmd to cmd & "  >/dev/null 2>&1 &" --   dump response, dont wait =  backgrund job,
   do shell script cmd
   return
end setCameraLevels



Code: Select all

#!/usr/bin/env python
## above needed to be executable without calling py

## ----- CODE (C) ------

import urllib2
import sys

## call eg:   /path to this file/setCameraLevels.py 1 1 1 1 0 '192.168.1.71' 

## --> how to call in indigo scheduled action: (without the ##)
## call executable python scipt from phyton, send output to nirwana (>..) +    do not wait until is finished (&)  otherwise indigo times-out
## this is day time setting, change  0 0 0 (bright. Contr. Satur.)   to more sensitve numbers for night times
##os.system ("'/Library/Application Support/Perceptive Automation/Indigo 6/Scripts/Attachments/setCameraLevels.py'  0 0 0 1 0 '192.168.1.71' >/dev/null 2>&1 & ")
##os.system ("'/Library/Application Support/Perceptive Automation/Indigo 6/Scripts/Attachments/setCameraLevels.py'  0 0 0 1 0 '192.168.1.73' >/dev/null 2>&1 & ")
##os.system ("'/Library/Application Support/Perceptive Automation/Indigo 6/Scripts/Attachments/setCameraLevels.py'  0 0 0 1 0 '192.168.1.74' >/dev/null 2>&1 & ")
##os.system ("'/Library/Application Support/Perceptive Automation/Indigo 6/Scripts/Attachments/setCameraLevels.py'  0 0 0 1 0 '192.168.1.75' >/dev/null 2>&1 & ")


Brightness525= int(sys.argv[1])
Contrast525= int(sys.argv[2])
Saturation525= int(sys.argv[3])
Antiflicker01= int(sys.argv[4])
MirrorOnOff01= int(sys.argv[5])
theIPnumber = sys.argv[6]
theUserid="xxx"
thePassword="xxx"

## map input(-5 ..+5)    -5--> 1 , .. , +5 --> 128         levels on webpage
map = ["1", "14", "26", "39", "51", "64", "77", "90", "103", "115", "128"]
   
## check input ranges
if Brightness525 >= 5: Brightness525 = 5
if Brightness525 <= -5: Brightness525 = -5
setBrightness = map[(Brightness525 +5)]
   
if Contrast525 >= 5: Contrast525 = 5
if Contrast525 <= -5: Contrast525 = -5
setContrast = map[(Contrast525 +5)]
   
if Saturation525 >= 5: Saturation525 = 5
if Saturation525 <= -5: Saturation525 = -5
setSaturation = map[(Saturation525 +5)]
   
if Antiflicker01 >= 1: setAntiflickerEnable = '1'
if Antiflicker01 <= 0: setAntiflickerEnable = '0'
   
if MirrorOnOff01 >= 1: setMirrorOnOff = '1'
if MirrorOnOff01 <= 0: setMirrorOnOff = '0'

## compile data string to be send to URL
theData = "&ReplySuccessPage=image.htm"
theData = theData + "&ReplyErrorPage=errrimg.htm"
theData = theData +"&BrightnessControl="+ setBrightness
theData = theData +"&ContrastControl="+setContrast
theData = theData +"&SaturationControl="+setSaturation
theData = theData +"&AntiFlickerEnable="+setAntiflickerEnable
theData = theData +"&Mirror="+setMirrorOnOff
theData = theData +"&ConfigSystemStream=Save"

theURL = 'http://'+theIPnumber+'/setSystemStream'


passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, theURL, theUserid, thePassword)
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))
response = urllib2.urlopen(theURL,theData)
exit()




Well, you'll need to install pycurl. I don't know where exactly you'd get it, but usually Python modules are delivered either as .egg files or with a setup.py file that you run to install it...

It seems this issue has been by-passed. But, I thought I'd comment anyway... pycurl is available from http://pycurl.sourceforge.net However, note that, at least for MacOS, it is only available in source code. This means you need to compile the module. That is not, per-se, a big problem for a one-off application. However, should you wish to use pycurl in a publicly distributed plugin, you'd have a nice bit of work including versions for PPC, Intel-32 & Intel-64 architectures and then making sure the right version was loaded.

OTOH, curl itself is great in shell scripts (and from AppleScripts via the shell).

On my learning voyage...

(C) Phyton solution using urllib. see code (C)
... ...
- launch py from within py and use & to send to background

comments welcome, especially for the PY section..

Thanks for sharing your work. Just two comments...

1. The use of & at the end of command to cause the command to execute in the background* is a shell feature (sh, ksh, bash, csh, etc.) and does not work in Python - where it is the binary AND operator. However, in Python, you could execute the command in its own thread to achieve the same effect.
2. For use in Indigo you could create the command as a function (updateLamp for example) and load it as an attachment in Indigo. Then, you could call it in an embedded script action (without needing the shell or the python sys module) like:
   
   updateLamp(Brightness525, Contrast525, Saturation525, Antiflicker01, MirrorOnOff01, theIPnumber, theUserid, thePassword)

*BTW, if you do use & in shell scripts, it is usually a good idea to redirect all output (stdout and stderr) to /dev/null. Otherwise, the command may not actually go into the background - especially when executed in an AppleScript "do shell script." For example:
some_command 1>/dev/null 2>&1 &