; in variable strings updated through http call

When the data in a http curl ... to update a string that contains a ";" it seems to cut off the string at that point.

curl ..... -d value={"rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab"} http://192.168.1.xx:8176/variables/varName [/code]
cuts it off at Rev 1.1;
sending it with a "," it works fine..

is this an indigo problem or curl?

Karl

is this an indigo problem or curl?

Neither, likely. It used to be recommended that a semicolon be a valid field separator in query string and some web servers still adhere to that. You can experiment by URL-encoding the semicolon (%3B).

Adam

I replaced the ";" with a ",".. thats fine by me

just had the question if this is an indigo restriction .. and where it happens: curl, transport, web receive, indigo , calling variableUpdated()

Karl

[jay (support)]

I'm confused about what you're actually executing - it looks like you're trying to post a dictionary in the value (which I don't believe is correct curl syntax for the RESTful API)? Can you post exactly what you're doing inside code tags so we can see what you're actually doing?

Code: Select all

cmd = ['/usr/bin/curl', '-u', u'xx:xx', '--digest', '-X', 'PUT', '-d', 'value={ "rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab"}', u'http://192.168.1.xx:xx/variables/pi_IN_Alive']
subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()
puts:
>>{ "rpi_type": "Pi 2 Model B Rev 1.1<<  into the variable "pi_IN_Alive")

replacing the ; in >>B Rev 1.1; ser#<<with a , makes it work

Karl

xx-ed the pwd .. ip numbers ...

[jay (support)]

Is you want in the variable this JSON string?

Code: Select all

{"rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab"}

Because if so the problem is that the string isn't quoted. If you want that JSON string inserted into a variable using curl, this should work:

Code: Select all

import urllib
put_string = urllib.quote('{ "rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab"}')
cmd = ['/usr/bin/curl', '-u', u'xx:xx', '--digest', '-X', 'PUT', '-d', 'value={}'.format(put_string), u'http://192.168.1.xx:xx/variables/pi_IN_Alive']
subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate())

I much prefer using the requests lib to using curl with subprocess since you're spawning another process (to do the curl command) which is more resource intensive than doing it completely within the python interpreter that's already running.

why does the >>;<< vs >>,<< behave differently? rest is the same.

Everything works if the string contains a coma but no semicolon.

this works: 'value={ "rpi_type": "Pi 2 Model B Rev 1.1, ser#3c8c7ab"}'
this does not 'value={ "rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab"}'

My guess is that Adam is correct.

[jay (support)]

If you don't quote it, then the semi-colon is used as a separator for post parameters (Adam is correct). If you quote it, then the entire thing is treated as a string and there's no attempt to break it apart.

urllib.quote() seems to work -- the ; make it through.
this:

Code: Select all

{"last_masterStart": "2018-12-17 23:11:12", "temp": "38.5", "rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab", "program": "master", "last_boot": "2018-12-17 02:14:39", "pi": "7", "ipAddress": "192.168.1.27", "op_sys": "8 (jessie); 4.4.50-v7+; #970 SMP Mon Feb 20 19:18:29 GMT 2017", "i2c_active": ""}

is converted to and send

Code: Select all

%7B%22last_masterStart%22%3A%20%222018-12-17%2023%3A11%3A12%22%2C%20%22temp%22%3A%20%2238.5%22%2C%20%22rpi_type%22%3A%20%22Pi%202%20Model%20B%20Rev%201.1%3B%20ser%233c8c7ab%22%2C%20%22program%22%3A%20%22master%22%2C%20%22last_boot%22%3A%20%222018-12-17%2002%3A14%3A39%22%2C%20%22pi%22%3A%20%227%22%2C%20%22ipAddress%22%3A%20%22192.168.1.27%22%2C%20%22op_sys%22%3A%20%228%20%28jessie%29%3B%204.4.50-v7%2B%3B%20%23970%20SMP%20Mon%20Feb%2020%2019%3A18%3A29%20GMT%202017%22%2C%20%22i2c_active%22%3A%20%22%22%7D

and then gets posted to the variables as

Code: Select all

{"last_masterStart": "2018-12-17 23:11:12", "temp": "38.5", "rpi_type": "Pi 2 Model B Rev 1.1; ser#3c8c7ab", "program": "master", "last_boot": "2018-12-17 02:14:39", "pi": "7", "ipAddress": "192.168.1.27", "op_sys": "8 (jessie); 4.4.50-v7+; #970 SMP Mon Feb 20 19:18:29 GMT 2017", "i2c_active": ""}

looks like any non letter or number gets converted (ie "':;,#_-+-{}[]) and then back again..

but now the string is 2 times as long.
As for cpu.. This runs on the RPIs and they run at ~ 5% cpu and the curl is done once every xx secs

When I have time will convert it to requests

thanks for the help

Karl

[jay (support)]

but now the string is 2 times as long.

If you're talking about what's PUTted to the RESTful API, that shouldn't matter (it's not long enough for it to be an issue on the IWS side).

If you're talking about what's stored in the Indigo variable, it's also not an issue but you can always just delete the keys you don't need before you create/send the JSON. If you're getting the JSON from somewhere else, then convert it to a Python dict, remove the keys you don't need, then convert it back to JSON before you send it. You can also compress the JSON a bit by removing the spaces after the key: part:

Code: Select all

json.dumps(my_dict, separators=(',',':'))

will remove the extra spaces.

Jay

thanks for the hint with the blanks, I was removing the spaces with .replace(" ",""), but only for non text data.

need to look more at the build in options.

Karl