Sensor value Pub/Sub Idea

Question to Matt and Jay and anyone who whats to chime in.

the subscribe method works fine for event type of messages eg sensor is up.

I am struggling with how/when to transmit / receive Meta data, ie which device state is participating, active, suspended , deleted ..

Here is what I would like to do:
ONE plugin will receive data from SEVERAL other plugins
in the plugin I am planning there will be menus where the user can select
1. ids of plugins that supply the information to be collected ("simple list of all active plugins minus - obviously not qualified for the task), this is "easy"
2. select individual devices from these plugins that are used to perform the functions

The second part is not . Where should the master data be managed - ie which devices/states participate in the send plugins or in the receiving plugin.

A) if the meta data management is done in the sending plugins they need to be able to tell in a managed way to the receiving plugin: here is my offer, take it or leave it-- no response required
-- issue: these message must be send when the receiving plugin is able to receive, so it has to be repeated in some frequency, doable
-- EACH of sending plugins will need to setup menus to manage which device/ state are participating -- menus: could be one central or a field in device edit for each device

B) if the receiving plugin is able to request the device/state participation there must be a 2 way message stream
1. receiving plugin sends a request, and 2. the sending plugins answer with the meta data and will setup the device/state event data stream.


Option A is easier to control but would require that each sending plugin adds a menu to manage the device/states that participate

Option B is easier for the user, one central place /menu where everything is managed, but the communication requirements become much more involved.
- either through action methods or through subscriptions, but that is much more involved, each sending plugin then needs to define which one it will partner with



So here my current thinking:
the receiving plugin does 90% of the work
1. in menu define sending plugins
2. in menu select devices from all indigo devices to participate
3. send action method requests to the sending plugins: I want data from this device/ state, please setup "indigo.server.broadcastToSubscribers(messageType, device/state data)"
4. will receive event data through subscriptions "indigo.server.subscribeToBroadcast(senderpluginId, messageType,device/state data)" for each sending plugin

the sending plugins will need to
1. setup receiving action method
- receive requests to add/remove device/.states to subscription
2. send broadcasts for the selected devices.

Q: does this make sense?

Karl

[jay (support)]

Hey Karl, I split this post off to a different topic since it's a worthwhile discussion but would likely get lost on the original thread.

I'm going to think a bit about it a bit more and then reply. My first impression is that what you've outlined seems very complex. I think I'd take a much narrower and more focused approach and figure out how to make it as simple as possible. KISS is always best for a first go at something IMO.

Jay,
thanks

if a plugin just dumps everything things are MUCH easier.

The complexity comes from considering:
I have about 5 device/state updates / sec ... how much would that tax a general broadcast/ subscription setup where plugins just BC everything?

if that is not an issue, yes then it is MUCH easier..

Karl

[jay (support)]

This subject was brought up in response to a question/comment about aggregating multiple presence plugins to get a better/more accurate presence state. My thinking is that we could create a high-level and easily implemented informal protocol that would make building such an aggregated device/service easier. Taking a high-level (and vertical) approach to this problem will help keep it simple (rather than trying to do everything).

In this scenario, you want any given plugin who understands some kind of presence to be able to broadcast out a message when presence changes. The majority of plugins that do presence detection present those as devices, so for the first iteration we could consider that as a given. You could then propose this informal specification:

message_type: presence_change
mesage_payload:

Code: Select all

{
   "namespace": "com.presenceprovider_a",
   "id": 123456,
   "status": "home"
}

In our initial scenario, "id" would be the device ID (or whatever uniquely identifies to the plugin the thing that's present or not) and "status" would be one of: "home", "away", "unavailable" (the latter meaning that for whatever reason the broadcaster doesn't know the current status). The subscriber would then be able to make whatever determinations it needs to when it receives one of these messages. Finally, "namespace" would be a unique identifier for who is providing the information - likely the plugin ID (see below for one use of this).

Here's an issue: when an aggregator starts up (or starts up a specific device) it's going to want to get immediate status updates. So we need some way for it to ask the provider for somewhat immediate status updates. Ideally, it would just be the aggregator broadcasting out a call for updates something like this:

message_type: presence_update_request
mesage_payload:

Code: Select all

{
   "namespace": "com.presenceaggregator",
   "updates_requested": {
      "com.presenceprovider_a": [123456]
   }
}

Presence Provider A would receive this broadcast, look for itself in "updates_requested" and if it finds a list then generate the appropriate presence_change broadcasts.

Unfortunately, our current pub/sub mechanism requires that you subscribe to a specific plugin (you can't subscribe to a specific message from any plugin). This isn't a problem for the presence aggregator as it can subscribe to messages from a plugin at the time that the aggregator device is started up. It is, however, an issue for presence providers, since they won't know about the aggregators out there. The ability to subscribe to a message regardless of where it comes from is definitely on the list of things to add.

In the meantime, however, there is another option that's pretty low-impact in terms of what the provider would have to add. Since plugins can call actions on other plugins, we can just specify a plugin action that a provider must supply in order to request immediate status. So, for instance, an action method like this could be implemented by the provider:

Code: Select all

def presenceUpdateRequest(self, action, dev, callerWaitingForResult=True):
    try:
      # Perform whatever logic to determine the state based on the dev passed in
      status = dev.states["status"]
      return status # "home", "away", or "unavailable"
   except:
      raise Exception("Couldn't process request")


and in your Actions.xml add:

Code: Select all

<Action id="presenceUpdateRequest" uiPath="hidden">
   <Name>Request Presence Update</Name>
   <CallbackMethod>presenceUpdateRequest</CallbackMethod>
</Action>


I think this is a good, relatively simple, start. Presence provider plugins would need to support two things:

• Publishing updates via the presence_change message described above
• In lieu of receiving a broadcast to update (since that doesn't work in the current implementation), implement the action above

This puts the logic for the high-level state in the provider plugin, where it should reside.

Aggregator plugins would need to:

• Create a device & UI for selecting devices that provide presence
• Subscribe to the presence_change message
• Call the presenceUpdateRequest action for each device when the aggregate device starts up

There are, of course, a TON of other things that can be done. But I think for a lightweight, high-level protocol this one is pretty nice and simple. KISS!

[jay (support)]

if a plugin just dumps everything things are MUCH easier.

The complexity comes from considering:
I have about 5 device/state updates / sec ... how much would that tax a general broadcast/ subscription setup where plugins just BC everything?

That is exactly why the solution needs to be a high-level one rather than a low-level one. That much data is not reasonable to pass around. Your plugin, which produces that much data, needs to aggregate it in a much higher level way. You can't really expect anyone else to understand/need that much data IMO. Your plugin is the one that should be making sense of all the data and create a much more basic representation of whatever it is the data represents. In terms of presence, it needs to render a "location" at a level that's usable.

I may have totally missed what it is you're trying to do. If so, please state the use case more clearly so we can understand what problem you're trying to solve.

Interesting reading this as I have built this functionality into a plugin that I wrote some time ago that I've not yet posted as an update. The concept is that you add multiple sources for presence detection into a single device that analyzes it based on criteria you add to determine if that object (person/thing) is present in the parameters.

The way I've implemented it is that it uses multiple detection objects such as motion sensors, cameras and switches to determine if a room is occupied at all based on the expected occupants. For instance, if only one occupant and motion is detected in room A but a switch was physically turned on in room B then room A is incorrect since room B required a physical presence to activate the switch).

I teamed that up with several geofencing options that I use, such as HomeKit and IFTTT to aggregate who should be home or not. Using date/time stamps for the data collected I am generally able to determine who is home and who is not and having multiple sources helps keep it redundant and more accurate.

With each person the algorithm gets quite complex and so I defined "areas", which are boundaries of objects that you group your various Indigo things into. The boundaries have borders that can be crossed and defined, so for instance you cannot enter my office without tripping two other presence detection devices (cameras/motion sensors) and then finally my office. Likewise you cannot leave the office without tripping these things, therefor I can detect with some measure of accuracy who is in what room.

Multiple occupancy detection is determined by when things happened and in what order. If those same three sensors trip for inbound to my office yet there was not a series of sensor trips for outbound then I know someone else entered my office while I am there.

It has been a very tricky plugin to build and it works most of the time (this how I know what lights to turn on and keep on and when to turn them off, we really don't really need to press buttons anymore). False positives absolutely happen in these cases because you have to rely on logical sensor data rather than a fixed detection method, such as a device everyone has to wear around the house. I try to mitigate that by having a series of "well, that can't happen" situations (i.e., you can't get from my office to the basement in less than 20 seconds unless you run, so if there was activity in the basement then either there is an intruder or my office detection failed and I really am in the basement).

I've worked on the plugin for years, it started out as a complex set of triggers in Indigo and migrated to a plugin and it's been quite complicated at times.

The single biggest reasons I haven't ever shared this plugin is because the UI is tricky (something I'm learning to deal with in this new HKB update as I develop new ways to work with tricky UI) and user support - too many moving parts and enough false positives that support would be nightmarish for someone who isn't pretty intuitive with home automation. I had planned to drop it earlier this year but in light of some of the crazy support request for HKB (i.e., people opening tickets and writing long posts and insisting the plugin is bugged only to respond to themselves a day or three later saying "oh yea, oops, that was my issue") I decided to hold off.

I don't know if this lends any insight into this topic but now you have it .

I don't know if this lends any insight into this topic but now you have it .

I’m very interested in your approach to this topic. In my thinking about it, the idea of device “rooms” with neighboring room information (akin to nodes in a linked list) seemed like a good approach. Also curious if you use probability, or even considered some accessible machine-learning.

Sorry for the tangent.


Sent from my iPhone using Tapatalk

here is what I have now:

BC from UNIFI, fingscan, pibeacon works as follows:
1. in config you can set enable BC for no dev, all, individual
2. in dev edit you can enable individual BC ie when this device changes status it will BC if general enable is setup all or individual
this gives the flexibilities to send everything or be picky

plugin "EVENTS" ( staring w existing code from fingscan which had this in its event feature, but somewhat convoluted -- one of my first plugins)
1. select devices that should participate, set timer parameters for immediate or delayed action
2. define EVENTS( = groups of devices eg guests, families, cars,..) with devices previously defined, set timers for immediate or delayed action
3. receives the BC and works the logic with timers to trigger events (oneHome, allHome, oneAway, allAway) of the event group
( it also has an action input channel, you can eg send the same message from a script:

Code: Select all

plug = indigo.server.getPlugin("com.karlwachs.events")
if not plug.isEnabled(): return 
plug.executeAction("receiveDeviceChangedAction", props ={"action": "event","id": 1791773772, "state":"status","newValue":"down","valueForON":"up"}) 


In addition when the EVENTS plugin receives unsolicited BC it will add the devices to its internal device lists, so that you don't have to define manually

In fingscan it also had distance to home from ifind, - but that plugin has gone another route..

will add some more parameters / props. that can be freely defined

currently the msg looks like:
[{'action': 'event', 'state': 'status', 'id': '1787475791', 'newValue': u'up', 'valueForON': 'up'}, {'action': 'event', 'state': 'status', 'id': '1897386914', 'newValue': u'expired', 'valueForON': 'up'}, {'action': 'event', 'state': 'status', 'id': '1345198519', 'newValue': u'down', 'valueForON': 'up'}]
a list of dicts with status changes.
id/state is the device-id/ state that has trigger the msg
newValue is the new value
valueForON is the value that indicates ON/UP/TRUE/AN/1 ..

It also manages the list of plugins it listens to from the devices selected to change the subscriptions to messages

in fingscan I was using regular actions to send info from pibeacon, unifi to fingscan that then as doing the EVENT stuff.

This is now much more flexible and add more options like distance , rooms ...

Yes this can become very complex, trying to make it as simple as possible for the user..

should have something posted this weekend .. will required upgrades in pibeacon,fingscan,unifi and adding "EVENTS" plugin

Karl

[jay (support)]

I don't see this as simple in any way. Guess I wasn't following what you were trying to accomplish...

it seems to work well. have in the EVENT plugin options to listen to certain plugin (with on & off = restart) and select devices / states only from these plugins, and setup events with a group of devices

Then in the sending plugin there are:
1. config option to switch BC off/all on/ on if device prop is enabled (device edit one prop)
2. in code ~ 15 lines to manage the sending with selection of devices . but only for on/ off states
actually not that complicated.

Load on the server is minimal, did not see any increase still ~ 5% total with video

anyone ready to test this?
you would need to upgrade plugins: pibeacon, fingscan, unifi if you use any of them
and install EVENETS plugin.
Should post something by tomorrow.

left to do
- logging of events (for debugging and info)
- adding variables for each event (oneHome ..)

for the future I would like to add something with doors: if door opens start timer window to enable home/away status change if ibeacon / iPhone are arriving / leaving.
w/o open/close of door any home away status change would be disabled.
That requires some tuning of timings , ie how long the window before and after door open/close to set arriving or leaving...

But that looks really promising. It would take care of iPhone going to sleep and then triggering away.. w/o door open/close no away event . that sounds logical and I don't see any obvious scenario where that should not work - may be if someone else is leaving while your phone goes to sleep ... but that time window is < ~2 minutes... somewhat unlikely

Karl