Using MQTT with Amazon IoT Broker

Greetings Joe!
First off, thanks for making this plugin!

I have a new request

I want to use Amazon's AWS IoT Core as my MQTT server, instead of running my own Mosquitto instance in an EC2 container. Why not, since I am already in AWS-land. Theirs is much more robust and much more secure. Plus, you get a lot of good metrics/stats/analytics in their portal right out of the box. Some other neat features, like Device Shadows, will come into play later...

Create an AWS Developer account if you don't have one.
https://us-east-1.console.aws.amazon.com/iot/home

Onboard a new Thing/Device that will represent Indigo. Pick your OS and Language - Linux/MacOS and Python in this case.
The important part of this step is to generate the X.509 certificates that will be associated with your "device" aka Indigo. These certificates will be required in order to get the AWS IoT MQTT client to talk to your personalized AWS IoT MQTT endpoint.

How to use the AWS IoT python SDK:
https://docs.aws.amazon.com/greengrass/ ... 17b7b1b3b3

The AWS IoT python SDK:
https://github.com/aws/aws-iot-device-sdk-python

A sample client that publishes "hello world" using the AWS IoT python SDK:
https://github.com/aws/aws-iot-device-s ... cPubSub.py

The good news is that their MQTT client class uses Paho under the covers. What we would need then, is the ability to tell your plugin that we want to use AWS IoT, so that you can invoke the right client class behind the scenes. We get our host value from the AWS IoT portal, linked above, use the left nav and click on Settings. That page will give you your personalized MQTT endpoint - something like "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com". Second, we need the plugin to be able to point to and make use of the generated X.509 certificates. These are created when you Onboard a new Thing in the AWS IoT portal. We would download them to the Indigo machine and point your plugin at them.

Does all that make sense?

Thank you for your consideration!
Regards,
Matt

Go to the GitHub issues page for this plugin: https://github.com/FlyingDiver/Indigo-MQTT/issues

Create an enhancement request. Put all that info in there.

I need to use GitHub to track this kind of stuff, otherwise it gets lost in the forums.

From your writeup, the only thing I really need to do is put in a way to specify the location of the certificates, and use those when connecting to the server, right?

Will do, and Yes, that should be it.

Some way to indicate that a user wants to use AWS IoT as their provider. Maybe a checkbox for "Use AWS IoT"?
Then, assuming you're using the AWS client provided in the python SDK, we just need to enter host + port + paths to 3 cert files.

"Host" in AWS IoT terms is called an Endpoint, FWIW.

From their sample PubSub client, it looks like the default port for WSS is 443 and for HTTPS its 8883 (seems backwards to me, but what do I know?)

Issue is created in GitHub: https://github.com/FlyingDiver/Indigo-MQTT/issues/6

Regards,
Matt

Thanks for making this update Joe!

Getting the following:

Code: Select all

   MQTT Connector Debug            MQTT Connector: logLevel = 5, loopDelay = 0.1
   Started plugin "MQTT Connector 0.3.2"
   MQTT Connector                  Starting MQTT Connector
   MQTT Connector                  AWS IoT Core: Starting Device
   MQTT Connector Threaddebug      AWS IoT Core: Device is current version: 0
   MQTT Connector Debug            AWS IoT Core: Broker __init__ address = axxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
axxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
:8883, ca_bundle = /Library/Application Support/Perceptive Automation/Indigo 7.4/aws-root-ca.crt, cert_file = /Library/Application Support/Perceptive Automation/Indigo 7.4/indigo-aws-certificate.pem.crt, private_key = /Library/Application Support/Perceptive Automation/Indigo 7.4/indigo-aws-private.pem.key
   MQTT Connector Error            AWS IoT Core: Exception while creating Broker object
Traceback (most recent call last):
  File "/Library/Application Support/Perceptive Automation/Indigo 7.4/Plugins/MQTT Connector.indigoPlugin/Contents/Server Plugin/aiot_broker.py", line 54, in __init__
    self.aIoTClient.connectAsync(ackCallback=self.onConnect)
  File "/Library/Application Support/Perceptive Automation/Indigo 7.4/Plugins/MQTT Connector.indigoPlugin/Contents/Server Plugin/AWSIoTPythonSDK/MQTTLib.py", line 545, in connectAsync
    return self._mqtt_core.connect_async(keepAliveIntervalSecond, ackCallback)
  File "/Library/Application Support/Perceptive Automation/Indigo 7.4/Plugins/MQTT Connector.indigoPlugin/Contents/Server Plugin/AWSIoTPythonSDK/core/protocol/mqtt_core.py", line 223, in connect_async
    raise e
gaierror: [Errno 8] nodename nor servname provided, or not known


Do I need to prefix my Endpoint with a protocol? I just entered it as:

Code: Select all

axxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com

Matt

There's something funky in your config.

Code: Select all

   MQTT Connector Debug            AWS IoT Core: Broker __init__ address = axxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
axxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com
:8883, ca_bundle = /Library/Application Support/Perceptive Automation/Indigo 7.4/aws-root-ca.crt, cert_file = /Library/Application Support/Perceptive Automation/Indigo 7.4/indigo-aws-certificate.pem.crt, private_key = /Library/Application Support/Perceptive Automation/Indigo 7.4/indigo-aws-private.pem.key


This looks like you have the Endpoint entered twice with a newline between them. Clear out that field and re-enter.

Joe

Derp, I couldn't figure out why my copy paste Shared clipboard on Screen Share was going wonky on me... it was pasting but just hiding the new lines out of field view...

Now looking at:

Code: Select all

   MQTT Connector Debug            MQTT Connector: logLevel = 5, loopDelay = 0.1
   Started plugin "MQTT Connector 0.3.2"
   MQTT Connector                  Starting MQTT Connector
   MQTT Connector                  AWS IoT Core: Starting Device
   MQTT Connector Threaddebug      AWS IoT Core: Device is current version: 0
   MQTT Connector Debug            AWS IoT Core: Broker __init__ address = axxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com:8883, ca_bundle = /Library/Application Support/Perceptive Automation/Indigo 7.4/aws-root-ca.crt, cert_file = /Library/Application Support/Perceptive Automation/Indigo 7.4/indigo-aws-certificate.pem.crt, private_key = /Library/Application Support/Perceptive Automation/Indigo 7.4/indigo-aws-private.pem.key
   MQTT Connector Debug            AWS IoT Core: Client is OffLine


Debug text is green, but the device is Offline (red).

Matt

What is the Client ID that your script is using in the connection to AWS IoT Core? It has to be referenced in the AWS IoT Policy that gets attached to the declared Indigo "device" or "thing" in your AWS IoT console.

Thanks,
Matt

What is the Client ID that your script is using in the connection to AWS IoT Core? It has to be referenced in the AWS IoT Policy that gets attached to the declared Indigo "device" or "thing" in your AWS IoT console.

Thanks,
Matt

Code: Select all

            self.aIoTClient = AWSIoTMQTTClient("indigo-mqtt-{}".format(device.id), useWebsocket=False)


My policy is:

Code: Select all

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}


Which doesn't require the device to be specified.

Ok, so if device.id is the id of the declared Indigo MQTT broker device, and if that device.id value is 123456789 ... then my client ID is indigo-mqtt-123456789

It is generally against best practice in an AWS security context to have a wide open policy. No big deal for testing certainly. I am trying to secure my instance a little better, so I will use the client ID computed from the Device ID.

Also, my Broker device says it is OffLine, but all the debug text from the Event Window is green (no errors that I can see). Is there a log file or a more verbose debug setting I can turn on to see why the device doesn't come Online?

Thanks!
Matt

Ok, so if device.id is the id of the declared Indigo MQTT broker device, and if that device.id value is 123456789 ... then my client ID is indigo-mqtt-123456789

Correct. Using the Device ID is the only way I can guarantee a unique client ID. Is it possible to use "indigo-mqtt-*" in the policy document?

It is generally against best practice in an AWS security context to have a wide open policy. No big deal for testing certainly. I am trying to secure my instance a little better, so I will use the client ID computed from the Device ID.

Yeah, I'm doing doing any production with this, that's for sure.

Also, my Broker device says it is OffLine, but all the debug text from the Event Window is green (no errors that I can see). Is there a log file or a more verbose debug setting I can turn on to see why the device doesn't come Online?

Did you try restarting the device or restarting the plugin? I set the status to "OnLine" when I get the "onConnect" callback, which is this message:

Code: Select all

        self.logger.debug(u"{}: Client Connected, mid = {}, rc = {}".format(device.name, mid, rc))


I don't think there is a way to declare a wildcard in the Policy for the client ID, but that is not a big deal. Another possibility would be to create a field in the Settings for letting the user set their Client ID of choice. Again, not too big of a concern at this point so long as I can know what the computed Client ID will be and set that value in my Policy if I want the added security.

I have been restarting the Indigo server to test any changes I've made. Upon startup, I see the log message above.

I created a random action to publish a random device payload:

Code: Select all

Action Group                    Publish Device Data
   MQTT Connector Threaddebug      AWS IoT Core: publishDeviceAction indigo/device/office_desk_lamp: {'states': [{'name': u'onOffState', 'value': False}], 'model': u'OutletLinc', 'address': u'0F.EF.FA', 'name': u'office desk lamp', 'deviceId': 211840855}, 0, 0
   MQTT Connector Debug            AWS IoT Core: Publishing to: indigo/device/office_desk_lamp (0), payload = {"states": [{"name": "onOffState", "value": false}], "model": "OutletLinc", "address": "0F.EF.FA", "name": "office desk lamp", "deviceId": 211840855}


But nothing comes through. As would be expected if my Broker Device is OffLine... Hmmm...

Matt

I spoke too soon! The device shows as OnLine. But when I try to execute my Action to publish device data I get:

Action Group Publish Device Data
MQTT Connector Threaddebug AWS IoT Core: publishDeviceAction indigo/device/office_desk_lamp: {'states': [{'name': u'onOffState', 'value': False}], 'model': u'OutletLinc', 'address': u'0F.EF.FA', 'name': u'office desk lamp', 'deviceId': 211840855}, 0, 0
MQTT Connector Debug AWS IoT Core: Publishing to: indigo/device/office_desk_lamp (0), payload = {"states": [{"name": "onOffState", "value": false}], "model": "OutletLinc", "address": "0F.EF.FA", "name": "office desk lamp", "deviceId": 211840855}
MQTT Connector Debug AWS IoT Core: Client is OffLine
MQTT Connector Debug AWS IoT Core: Client is Online
MQTT Connector Debug AWS IoT Core: Client Connected, mid = CONNECTED, rc = 0

Alrighty, I got it working by using the generic Policy you posted above. Something too restrictive in my Policy otherwise. I can hack at that and figure out whats going on.

Got it, so to declare wild cards in your topics within your Policy, use * not #, like so:

Code: Select all

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": "arn:aws:iot:us-east-2:xxxxxxxxxxxx:topic/indigo/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Subscribe"
      ],
      "Resource": "arn:aws:iot:us-east-2:xxxxxxxxxxxx:topic/indigo/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Connect"
      ],
      "Resource": [
        "arn:aws:iot:us-east-2:xxxxxxxxxxxx:client/indigo-mqtt-123456789"
      ]
    }
  ]
}


Thanks!
Matt

Ok I got it going mostly... just a couple last questions.

1) The default Device Publish Template is not producing valid JSON. Single quotes... vs double.... vs none. I will play with the format and report back what works. The AWS IoT console has a test client and when it receives messages, it's complaining that incoming messages are not JSON.

2) How do I Publish Device Data for all devices by default? If I add every device to the Publish list, that will work. If I use the Excludes checkbox and don't add any devices to the list, I would expect it to publish a change event for every device (since none are excluded)... Is that not how that works?

This is really rad otherwise! Thanks Joe

Matt

I split this topic off from the other thread.