Request assist with first MQTT project.

I'm trying to leverage an app "Owntracks" Full documentation at https://owntracks.org/booklet/guide/topics/

Background: Life360 is not reliable. FFM is now limited by apple's family size. This looks like a great alternative for presence, location tracking, etc and looks to be a lot more secure and was built for tinkerers like us.

1) Set up MQTT Broker Plugin
2) Set up MQTT Connector Plugin
3) Set up MQTT Shims Plugin

So far I have it working at about 80% while I'm in the house.
I set up a generic shim device (My phone) that is getting all these cool states from the app: battery level, lattitude, longitude, timestamp, in regions (what geofence I'm in) etc.

Problem 1: Lattitude and longitude are getting pulled in as a real number (rounded two past the decimal). I kinda need the whole thing for this to be useful.

Problem 2: Security. Right now this is cool on my network using my indigo machine IP address. But I'll need to switch that to the reflector and then? Password? API key? and then figure out how to put that in the app so they keep talking to each other.

Problem 3: 2-way communication? I can repeat these steps for other phones in the house, but I think they are suposed to be able to share with each other using the Broker as the communication hub?

I was trying to wrap my head around the templates to see if maybe I could write a new one to define all the states that corespond with the app.... but I think more states will show up when I start entering and leaving and stuff.

Any pointers would be great. I can post screen shots or try to explain stuff better. Honestly, I'm suprised it worked at all given my lack of MQTT knowledge.

Edit 1/15: Finally got an external MQTT broker running on a spare raspberry pi3b. (Side note, that 3b has dramatically gone up in value since I purchased it for $34).
With the MQTT Connector and MQTT Shims, I was able to create a device for each cell phone running the Owntracks app. I was also able to use the ghostxml plugin to pull historical data from the "recorder" part of the app that lives on the raspberry pi. It's not pretty, but I just made that happen earler today.

With the MQTT broker running solid, I'm trying a different approach. There is a Home Assistant image available on the Raspberry Pi Imager app (MacOS). So far so good. A few clicks here and there, no terminal and I have Home Assistant running with Mosquito Broker. Why this route? It might offer other integration options for presence detection, I need a broker anyway and I may need to control stuff on site that is out of range from my indigo network. I'll tinker with this for a day or so to decide if it's worth keeping or not.

Problem 1: Lattitude and longitude are getting pulled in as a real number (rounded two past the decimal). I kinda need the whole thing for this to be useful.

Enable debug logging on the CONNECTOR plugin, and then post one of the messages from one of the devices. That will let us see the exact contents of the MQTT topic and payload.

Problem 2: Security. Right now this is cool on my network using my indigo machine IP address. But I'll need to switch that to the reflector and then? Password? API key? and then figure out how to put that in the app so they keep talking to each other.

You CANNOT run MQTT over the reflector. Wrong port, wrong protocol. You need to have the MQTT broker accessible to all the devices, all the time, which means accessible from the open Internet. The MQTT Broker plugin is NOT recommended for this. I'm pretty sure I stated that in the readme for the plugin. So you would need to run a fully featured MQTT server somewhere. If that's on a machine on your local LAN, then you'll need to do port forwarding from your router.

Problem 3: 2-way communication? I can repeat these steps for other phones in the house, but I think they are suposed to be able to share with each other using the Broker as the communication hub?

I'd have to look at the documentation for the app. But if the app is subscribing (receiving) all the same messages it's sending, then yes each app (device) would have the info on all the other devices.

I was trying to wrap my head around the templates to see if maybe I could write a new one to define all the states that corespond with the app.... but I think more states will show up when I start entering and leaving and stuff.

The templates are only used when you're using the Indigo controls to send commands to the MQTT-connected devices. I don't think that applies in your case.

There's a menu command for the Shim plugin - "Write Device Template to Log". Use that on the Shim device you created, then post the output here.

Enable debug logging on the CONNECTOR plugin, and then post one of the messages from one of the devices. That will let us see the exact contents of the MQTT topic and payload.

Code: Select all

MQTT Connector Debug            MQTT Local: processReceivedMessage: owntracks/Bill/Bill_i13mini, payload: {"_type":"location","acc":5,"alt":8,"batt":100,"bs":1,"BSSID":"cc:40:d0:40:11:16","cog":109,"conn":"w","created_at":1705066749,"inregions":["ISL"],"inrids":["b39b40"],"lat":28.554298,"lon":-94.263228,"m":2,"p":100.675,"SSID":"VHM","t":"t","tid":"BM","tst":1705066348,"vac":3,"vel":5}
   MQTT Connector Debug            MQTT Match Bill Phone: type = topicMatch, broker = 286938242
   MQTT Connector Debug            MQTT Match Bill Phone: Matching complete, is_match = True

You CANNOT run MQTT over the reflector. Wrong port, wrong protocol. You need to have the MQTT broker accessible to all the devices, all the time, which means accessible from the open Internet. The MQTT Broker plugin is NOT recommended for this. I'm pretty sure I stated that in the readme for the plugin. So you would need to run a fully featured MQTT server somewhere. If that's on a machine on your local LAN, then you'll need to do port forwarding from your router.

I didn't see it in the readme, but I'm sure it's documented in here. (You try to make it idiot proof, then along comes a bigger idiot.) The owntracks documenation includes directions for that with examples to work with the app.... so I should be able pick a machine to use and muddle through it.

I'd have to look at the documentation for the app. But if the app is subscribing (receiving) all the same messages it's sending, then yes each app (device) would have the info on all the other devices.

It appears so. The app doesn't talk to an app server. It communicates between devices (for location sharing) and with the MQTT broker.

The templates are only used when you're using the Indigo controls to send commands to the MQTT-connected devices. I don't think that applies in your case.

There's a menu command for the Shim plugin - "Write Device Template to Log". Use that on the Shim device you created, then post the output here.

You can send commands for requesting a location update. (cmd/action/reportLocation) From the app, you can adjust when or how that the app sends location updates. (Quiet/Manual/Significant/Move)
I also thought if I used a template, I could ensure all the states are present (and stay present) and make it easier if someone down the line want's to try using this app with shims.

Code: Select all

message_type: '##billPhone##'
props:
    SupportsBatteryLevel: true
    SupportsStatusRequest: true
    battery_payload_key: batt
    custom_decoder: /Library/Application Support/Perceptive Automation/Indigo 2023.1/../Python3-includes/MQTT Shims Decoders/Expand.py
    state_dict_payload_key: .
    state_location: payload
    state_location_payload_type: json
    status_action_payload: reportLocation
    status_action_template: action
    uid_location: topic
    uid_location_topic_field: '2'
trigger:
    match_list: '["Match: owntracks", "Match: Bill", "Any: ", "End: "]'
    queueMessage: true
type: shimGeneric

And this is what the device looks like so far:

Screenshot 2024-01-12 at 7.55.36 AM.png (36.18 KiB) Viewed 1069 times

Take the "End" term out of the trigger. I think the device sends additional topics you might want to capture.

I'll need to look at the code to see what's going on with the lat/long values. The JSON payload has the full values. I might have a bug in the plugin when storing that data to the states.

Oops. My bad. I'm actually truncating floats to 2 decimal places in the states. Not sure why I did that. I'll roll a fix for that today.

Oops. I actually can't release an update for the plugin right away. It's currently got changes in it for the next release of Indigo and it would be more work than I want to do to roll it back and make this change. It'll have to wait.

Don't rush on my account. I'm still looking for a three sided coin to flip to decide where to install an MQTT Broker (on my indigo mac, on my QNAP, or on a dedicate Raspberry Pi (leftover from a Pi Beacon experiment... worked great, but no buy-in... couldn't get anyone else in the house to carry around a beacon.... but really good at telling me where in the house I am.)

That, by the way, is why I'm going this route.... you have to pry to get a phone away from a teenager.

You CANNOT run MQTT over the reflector. Wrong port, wrong protocol. You need to have the MQTT broker accessible to all the devices, all the time, which means accessible from the open Internet. The MQTT Broker plugin is NOT recommended for this. I'm pretty sure I stated that in the readme for the plugin. So you would need to run a fully featured MQTT server somewhere. If that's on a machine on your local LAN, then you'll need to do port forwarding from your router.

That was an all day experience. Decided to follow your example with the docker thing on my mac. Well, my indigo mac is about 10 years old, just kinda limping along at this point and it wouldn't recognize the docker.dmg file (kept saying it was corrupted). Okay, scratch that, lets go the brew route. Nope. That refused to work as well.

Ended up re-setting up one of my old raspberry pi devices. Fast forward 10 hours and the dang thing is finally running... DNS updated, port forwarded, changed up the Connection broker and trigger and my shim device is updating again. I'm going to bed and if that thing isn't still running in the morning, I'll either start over, or grab a ball peen hammer and pound my nuts flat.

Oops. I actually can't release an update for the plugin right away. It's currently got changes in it for the next release of Indigo and it would be more work than I want to do to roll it back and make this change. It'll have to wait.

I noticed the change in your lates update. It's much appreciated. I can now see the full Lat/Long for my devices. Thank you.

Thank you again for your last update... now I have a new problem....

The configuration for the MQTT broker changed.... so the connection is different. I can connect from the companion phone app without issue. I even figured out how to connect via MQTT Explorer.... but I can't get the plugin device configured to connect.

The old setting: MQTT://rPi_IP_Address:1883 username / password Encryption off.

The new setting: MQTT://rPi_IP_Address/owntracks:8883 username / password Encryption on.

I'm seeing the traffic on MQTT Explorer with the new settings... on indigo I get:

Code: Select all

   MQTT Connector Debug            ISL_MQTT: Device is current version: 0
   MQTT Connector Debug            ISL_MQTT: Broker __init__ address = rPi_IP_Address/owntracks, port = 8883, protocol = 3, transport = tcp
   MQTT Connector Debug            ISL_MQTT: No cert file provided, using default cert_file
   MQTT Connector Debug            ISL_MQTT: do_connect
   MQTT Connector Debug            ISL_MQTT: Broker connect error: [Errno 8] nodename nor servname provided, or not known

Any advice?

That error means the name or IP address can’t be resolved. So double check what you have in the hostname field.


Sent from my iPhone using Tapatalk

What is that “owntracks” term? You can’t put a url in that field.


Sent from my iPhone using Tapatalk

I was experimenting with different things....
The IP address didn't change from before.

Once I upgraded the software on the raspberry pi, the instance of MQTT is now configured by the owntracks setup.

The difference on the MQTT side, different set of usernames and passwords, port changed from 1883 to 8883 and now it uses TLS.

On the MQTT Explorer application, I made those changes and I can connect to the broker.
On Indigo, I made those changes, and I get an error.

Putting it back to just the URL, now I get a ssl certificate error.
MQTT Connector Debug ISL_MQTT: Device is current version: 0
MQTT Connector Debug ISL_MQTT: Broker __init__ address = 192.168.1.86, port = 8883, protocol = 3, transport = tcp
MQTT Connector Debug ISL_MQTT: No cert file provided, using default cert_file
MQTT Connector Debug ISL_MQTT: do_connect
MQTT Connector Debug ISL_MQTT: Broker connect error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '192.168.1.86'. (_ssl.c:1006)

If I remember correctly (from something I read that probably you posted) there is a certificate something on the mosquito side that maybe I can copy onto my indigo machine and point to it from the plugin?

There's really no value to having SSL enabled if it's all on your local LAN. But if you are using SSL, you need a certificate and there's a place in the connection config dialog to specify the path.

I tried both.
On the indigo side, if I turn off SSL, I get

Code: Select all

  MQTT Connector Error            ISL_MQTT: Disconnected with result code 7

Following the mosquito cofig file, it points to:
certfile /usr/local/owntracks/tls/cert.crt
keyfile /usr/local/owntracks/tls/cert.key

I copied "cert.crt" to the indigo directory under a new folder "MQTT_Cert", pointed to it from the indigo device:

Code: Select all

   MQTT Connector                  ISL_MQTT: Starting Device
   MQTT Connector Debug            ISL_MQTT: Device is current version: 0
   MQTT Connector Debug            ISL_MQTT: Broker __init__ address = 192.168.1.86, port = 8883, protocol = 3, transport = tcp
   MQTT Connector Debug            ISL_MQTT: Specified cert_file 'MQTT_Cert/cert.crt'
   MQTT Connector Debug            ISL_MQTT: Using cert_file '/Library/Application Support/Perceptive Automation/Indigo 2023.2/MQTT_Cert/cert.crt'
   MQTT Connector Debug            ISL_MQTT: do_connect
   MQTT Connector Debug            ISL_MQTT: Broker connect error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)

Outside of this conversation, I'm trying to figure out how to have a non-SSL local path. My thought is to just add "listener 1883" to the config file. My only concern with that is 1883 is already in the config file as a loopback "listener 1883 127.0.0.1". Next thought, add 1882 or 1884?