MQTT Gateway Plugin

Posted on
Wed Oct 07, 2015 5:02 pm
jeremyjjr offline
Posts: 104
Joined: Sep 10, 2013
Location: St Albans, UK

MQTT Gateway Plugin

Hi all,

I’ve been working on a number of projects that use the MQTT protocol to publish device state changes and have been using a Mosquitto Broker running on my Indigo server for quite a while now. I’ve just finished development of the v1 release of my MQTT Gateway for Indigo. The plugin allows you to create a set of 'Sensor' devices that correspond to specific Topics on the Broker. Threading is used to support multiple Topics with dedicated listeners.

Features

1. Create an Indigo device for each MQTT topic that you want to subscribe to
2. If an ‘On’ or ‘Off’ is broadcast on the specific topic, the device onOffState is updated
3. Any other message broadcast on the same topic will result in the topicMessage value being updated
4. Supports Triggers and Status Requests (your corresponding MQTT device will need to implement the response)
5. Provides an Action for sending any message to the subscribed topic

Pre-requisites

A working MQTT Broker supporting MQTT v3.1. I've used and tested this build with Mosquitto 1.4. Visit the following link for instructions to set one up if you need some guidance - http://wp.me/p6M2U5-3q

Download

Version 1.1.1 can be found here:
https://github.com/jeremyjjr/mqttGateway

Hope you find it useful and easy to use.
Last edited by jeremyjjr on Wed Jun 14, 2017 5:22 pm, edited 10 times in total.

Posted on
Wed Oct 07, 2015 6:32 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: MQTT Gateway Plugin

Hi Jeremy,

Very cool – thanks for sharing this. I'm curious, are you using this to integrate physical hardware (sensors) via MQTT, or are you using it to integrate some other services/messaging?

Image

Posted on
Thu Oct 08, 2015 1:15 am
jeremyjjr offline
Posts: 104
Joined: Sep 10, 2013
Location: St Albans, UK

Re: MQTT Gateway Plugin

Hey Matt,

Good question. My current project has an external CCTV camera set up to detect motion within a given area. Notification from the CCTV system is limited to e-mail. When motion is detected the CCTV DVR sends an e-mail to a local PostFix SMTP server running on a Raspberry PI. There is a PostFix Filter inspecting e-mails prior to delivery to the local inbox and it watches for e-mails from the CCTV DVR with specific content. This is essentially acting as an e-mail to MQTT bridge.

When a match is found, the filter raises a MQTT message saying motion has been detected (or 'On'). A timer is also started to count 20 seconds and when elapsed a second MQTT message is issued closing the motion event (or 'Off'). The CCTV system is configured to send motion alerts every thirty seconds, so another e-mail will arrive 10 seconds later if there is continued motion. This effectively enables a 'PIR' function to be added to the CCTV solution with the added benefit of detecting motion in very specific areas. A matching sensor device inside Indigo is now able to immediately notify of movement and issue triggered actions when turned on. For example - moving the blinds or turning on a light to give the sense of 'presence' within the home.

All in all, this works very well. I set up the e-mail to MQTT bridge as I needed instantaneous notification and response to an event and regular e-mail scanning just wasn't fast enough. I'm writing up the project through a series of blog posts over at http://simplifiedthinking.co.uk/blog and they should get loaded over the next few days.

I'm now considering using another RaspberryPI as a Beacon to see how I can better detect specific devices within a room or zone and MQTT will again be the ideal method for sending notifications around the house. Let's see how that goes!

J.

Posted on
Thu Oct 08, 2015 8:43 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: MQTT Gateway Plugin

Cool – keep us posted. I like your seven levels of automation article BTW.

Image

Posted on
Mon Nov 23, 2015 2:20 pm
rlust offline
Posts: 93
Joined: Jan 12, 2009
Location: Ohio

Re: MQTT Gateway Plugin

Any thought to using this with OWNTracks to track your location relative to home?

I want to use this as a location trigger for home and away.

Please let me know your progress.

Thanks,
Randy

Posted on
Mon Nov 23, 2015 3:59 pm
jeremyjjr offline
Posts: 104
Joined: Sep 10, 2013
Location: St Albans, UK

Re: MQTT Gateway Plugin

Hi,

For location awareness I use the Pyicloud library available on GitHub. My first development in Indigo was an iPhone device which uses this library at its core. This works better than MQTT for my needs as I've not opened my MQTT broker to the outside world.

All the best. J.

Posted on
Tue Dec 01, 2015 9:20 am
rlust offline
Posts: 93
Joined: Jan 12, 2009
Location: Ohio

Re: MQTT Gateway Plugin

MQTT gateway error

I have had good success with using this plugin with Owntracks over the past week.

Recently I started having the following error with the plugin.

If I stop the plugin then restart it will work for a while.

mqttGateway Error Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
File "plugin.py", line 88, in runConcurrentThread
File "plugin.py", line 99, in deviceStartComm
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 616, in __init__
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 978, in _get_handles
<type 'exceptions.OSError'>: [Errno 24] Too many open files

mqttGateway Error plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
mqttGateway Error Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
File "plugin.py", line 88, in runConcurrentThread
File "plugin.py", line 99, in deviceStartComm
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 616, in __init__
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 978, in _get_handles
<type 'exceptions.OSError'>: [Errno 24] Too many open files

Thanks for any help or advice.

Randy

Posted on
Tue Dec 01, 2015 9:54 am
jeremyjjr offline
Posts: 104
Joined: Sep 10, 2013
Location: St Albans, UK

Re: MQTT Gateway Plugin

Hi Randy,

Glad to hear you're making progress ...

Sounds like the background subprocesses are either failing (and not being trapped) or restarted over and over ... hence created too many open 'files'. There is a debug mode available under the plugin settings. Could you enable that and then capture the log data (including startup, good running and the failure point). If you could take a screen capture of the settings of the device you've created that would also help with me looking at the problem.

Thanks,

Jeremy.

Posted on
Tue Dec 01, 2015 4:21 pm
rlust offline
Posts: 93
Joined: Jan 12, 2009
Location: Ohio

Re: MQTT Gateway Plugin

Thanks her your help!
I have enabled logging and have the following error:

mqttGateway Debug Starting ConcurrentThread ... Active threads = 6
mqttGateway Error Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
File "plugin.py", line 88, in runConcurrentThread
File "plugin.py", line 99, in deviceStartComm
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 616, in __init__
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 978, in _get_handles
<type 'exceptions.OSError'>: [Errno 24] Too many open files

mqttGateway Error plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
Received X10 "Activity Kitchen" on


Here is the log prior to an error:

Started plugin "mqttGateway 1.0.1"
mqttGateway Debug startup called
mqttGateway Debug io_queue_reader started
mqttGateway Debug mqtt_listener for localhost:test started with pid: 78338
mqttGateway Debug mqtt_listener for home.rlust.net:owntracks/rc/kim started with pid: 78339
mqttGateway Debug mqtt_listener for localhost:light/state started with pid: 78340
mqttGateway Debug mqtt_listener for localhost:location started with pid: 78341
mqttGateway Debug mqtt_listener for home.rlust.net:owntracks/rc/test started with pid: 78342
mqttGateway Debug mqtt_listener for home.rlust.net:owntracks/rc/test/event started with pid: 78343
mqttGateway Debug mqtt_listener for localhost:device/status started with pid: 78344
mqttGateway Debug Starting ConcurrentThread ... Active threads = 9
mqttGateway Debug mqtt_listener for localhost:device/status has stopped
mqttGateway Debug mqtt_listener for localhost:test has stopped
mqttGateway Debug mqtt_listener for localhost:light/state has stopped
mqttGateway Debug mqtt_listener for localhost:location has stopped
mqttGateway Debug io_queue_reader:home.rlust.net:owntracks/rc/test: {"t":"t","tst":1449006038,"acc":165,"_type":"location","alt":272,"lon":-82.4705313932369,"vac":16,"p":97.4686279296875,"lat":40.06728377620956,"batt":47,"tid":"rc"}
mqttGateway mqtt owntracks RC received mqtt message from owntracks/rc/test
mqttGateway Debug io_queue_reader:home.rlust.net:owntracks/rc/test/event: away
mqttGateway mqtt owntracks RC Enter/Leave received mqtt message from owntracks/rc/test/event
mqttGateway Debug mqtt_listener for localhost:test started with pid: 78357
mqttGateway Debug mqtt_listener for localhost:light/state started with pid: 78358
mqttGateway Debug mqtt_listener for localhost:test has stopped
mqttGateway Debug mqtt_listener for localhost:location started with pid: 78359
mqttGateway Debug mqtt_listener for localhost:light/state has stopped
mqttGateway Debug mqtt_listener for localhost:device/status started with pid: 78360
mqttGateway Debug mqtt_listener for localhost:location has stopped
mqttGateway Debug mqtt_listener for localhost:device/status has stopped

Posted on
Wed Dec 02, 2015 3:40 am
jeremyjjr offline
Posts: 104
Joined: Sep 10, 2013
Location: St Albans, UK

Re: MQTT Gateway Plugin

So reviewing the log gives a couple of pointers. Firstly - your system process count is VERY high - this either means your server has been running a very long time or that a lot of new processes are being created. I'm going to assume the latter.

Version 1.0.1 of the plugin used a python array to maintain pointers to each sub process. However, when this sub process died, it did not clean up the array and therefore would end up with the array growing forever. In Version 1.0.2 (see download link in original post above) I've changed the data structure from an array to a list, ensuring that we will only ever have one entry in the list per device created in Indigo - this should get rid of the 'Too many open files' error ... I hope!

However, this is not the root cause of the problem - and there is something else going on in the logs that looks a little suspect. Once you've created a device in Indigo, the plugin launches a sub process in the background which starts the MQTT listener and feeds any received data back to the plugin. Your logs are showing that your listener sub processes are starting and stopping quite frequently - which isn't good. On my system here they run for weeks at a time without issue. Focusing on one specific failing device - the one that uses the topic 'device/status'. Try running the following command in a command line window on the same server that Indigo is running.

Code: Select all
mosquitto_sub -h localhost -t device/status


This command should run without issue and patiently wait until it receives a topic broadcast that it's interested in. If you run it in parallel with the listener in Indigo, they should both run in tandem. If the Indigo process stops and restarts whilst the manual command line continues to run, then there is another issue I'm not seeing and we should pick that up. However, my suspicion is that both commands will fail at roughly the same time which should then give you some more information to work on.

Let me know how you get on.

Posted on
Wed Dec 02, 2015 8:50 am
rlust offline
Posts: 93
Joined: Jan 12, 2009
Location: Ohio

Re: MQTT Gateway Plugin

I downloaded the plugin and it says it is the same version 1.01 as I have installed.

Posted on
Wed Dec 02, 2015 9:07 am
rlust offline
Posts: 93
Joined: Jan 12, 2009
Location: Ohio

Re: MQTT Gateway Plugin

I think I am finding the root of my issues.
I am running mosquitto on a different host
I have tried to install on my local machine but having problems.
I followed the instructions to install on local host but having issues.

if I try the following command on my local machine I get the following error
mosquitto_sub -t topic/state Connection refused


Maybe I need to clean out everything and start over.

Any good idea on how to do that on my mac?

Thanks for all of your help!

Posted on
Wed Dec 02, 2015 9:19 am
rlust offline
Posts: 93
Joined: Jan 12, 2009
Location: Ohio

Re: MQTT Gateway Plugin

I think I solved the problem!

I did a brew uninstall of mosquitto then reinstalled

Looks to be working with no errors at this point!!!

By the way the link to how to install mosquitto has a typo on 2 entries with only one t. :lol:

Posted on
Mon Dec 07, 2015 2:28 pm
afulki offline
Posts: 15
Joined: Jun 01, 2015

Re: MQTT Gateway Plugin

Thank you for the plugin.

I'm finding it very useful with my home automation setup. I am running a number of ESP8266 based modules (NodeMCU and Sparkfun Things) publishing and subscribing to topics on Mosquito running on the MacPro alongside Indigo. This nicely allows me to update state in variables etc. from MQTT.

If there was one thing I would like to see is an ability to trigger every message rather than only when the payload changes, not a big deal though.

Thanks again.
Gary

Posted on
Thu Dec 10, 2015 11:57 am
jeremyjjr offline
Posts: 104
Joined: Sep 10, 2013
Location: St Albans, UK

Re: MQTT Gateway Plugin

Gary,

I've updated to 1.0.4 and added a new device status called 'topicNotification' to the device model. Now, whenever a message is received this status toggles from '0' to '1' and back again even if the message doesn't change. Hopefully, this meets with your needs - let me know how it goes.

All the best,

J.

Who is online

Users browsing this forum: No registered users and 4 guests