Concurrent Thread Plugin level vs device level.

Posted on
Sun Nov 19, 2017 1:47 pm
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Concurrent Thread Plugin level vs device level.

Forgive my lack of general programming knowledge here. Maybe this answer is obvious, but it appears I need a bit of handholding...

So I've made a few plugins, all of a similar style. Serial-port interfaces where the plugin itself connects to another system via serial port, and then controls multiple devices through that, and the runConcurrentThread process can parse off responses into the correct state changes for each device, etc...

But can someone describe to me (in caveman terms here) how this works when the kind of 'Concurrent Thread' process is more Device-specific? i.e. if it was some sort of serial communications plugin, I'm thinking of where each 'device' has its OWN serial connection, so it's not all going through one main one, and that connection is defined at the device level. There's no 'runDeviceConcurrentThread' is there?

I'm thinking that I make one? (and then the main, runConcurrentThread isn't a loop, to do anything so much as a loop to just start the deviceConcurrentThread for each device that exits and is enabled? (and then I guess stopConcurrentThread maybe just passes a kill variable to each deviceConcurrentThread?) Or is there some simpler, more obvious way here?

(I know people have built plugins like this before, and I'm trying to go through them and look, but these answers aren't super-obvious to me).

Any help is appreciated.

http://nerdhome.jimdo.com

Posted on
Sun Nov 19, 2017 2:31 pm
kw123 offline
User avatar
Posts: 4971
Joined: May 12, 2013
Location: dallas, tx

Re: Concurrent Thread Plugin level vs device level.

runConcurrentThread is THE MAIN loop. it must run

do something like this
Code: Select all
def runConcurrentThread(self):
    try:
       while True:
            for ii in range(ofNumberofDevs):
                self.handleDev(ii)
            self.sleep(0.5)
    except: pass

Code: Select all
    def handleDev(self,ii):
  ## do you serial comm stuff here for dev# ii
       return


that's a very basic version.

you could loop of the devs instead of numbers with
Code: Select all
...
for dev in indigo.devices.iter("com.yourname.xxx" ):
   ## select the devices you like to manage
    self.handleDev(dev)
   ...

def handleDev(self,dev)
  ## do you serial comm stuff here for dev
       return


[/code]


you MUST use self.sleep() (not time.sleep() ). that allows indigo to restart the plugin in a nice way



Karl

Posted on
Sun Nov 19, 2017 5:38 pm
matt (support) offline
Site Admin
User avatar
Posts: 18070
Joined: Jan 27, 2003
Location: Texas

Re: Concurrent Thread Plugin level vs device level.

I don't think Karl understood quite what you were asking.

In a case where there are multiple blocking serial connections you will want each one to have its own python thread. You can use python's normal threading support to create and manage those threads.

Indigo does not have a runDeviceConcurrentThread() type of method. The runConcurrentThread() callback is optional (you can not define it). We just include it and have the thread automatically created in the internal plumbing (only if runConcurrentThread is defined) to simplify things a bit for plugin developers since 1 extra processing thread is often useful. But in this case it sounds like you want N threads, so you can just manually handle those and remove the runConcurrentThread definition.

Image

Posted on
Sun Nov 19, 2017 7:10 pm
kw123 offline
User avatar
Posts: 4971
Joined: May 12, 2013
Location: dallas, tx

Re: Concurrent Thread Plugin level vs device level.

I guess that was from one caveman to the other :P

Posted on
Sun Nov 19, 2017 8:10 pm
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Re: Concurrent Thread Plugin level vs device level.

Ok Matt, so I think that’s in line with how I thought this was going to work.

Mark my words - I’ll be back to this thread with more questions soon enough.

http://nerdhome.jimdo.com

Posted on
Tue Nov 21, 2017 7:55 am
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Re: Concurrent Thread Plugin level vs device level.

Ok, another question. This comes to me as a result the topic at hand, but if it needs to be its own topic, I can start a new thread...

I'm wondering how to scope a variable at the device-level if that makes sense.

Context:

So I basically created a method called runDevConcurrentThread(self, dev) (I assume I just call this at the end of the deviceStartComm procedure?)

Anyway, this thread will loop and handle the processing for each device. As a part of the processing, I created a bunch of little sub-methods that are basically data conversions and stuff. They will take a piece of data and convert it or do something else with it depending on its type. Sometimes these little sub procedures will change the state of a variable I defined outside of them. (Note: I built my first draft of this process as just a single device process, and in this case I just scoped these variables globally. I don't think that will work here, since there could be multiple devices and then multiple device threads would be using only one variable.

These variables would basically have the same scope as a device state, except I don't want to define them as states since they're not really of any use to the user, just sort of transient data for the plugin. (Maybe if states could be hidden or something, that would fit the requirements, but that seem inefficient on some level) For example, a variable might be like a timestamp of the last time any data arrived for the device (not necessarily a state change, just data). Now, my plugin has a use for this, the user does not and each device needs its own. Where would I define something like dev.timestamp? In deviceStartComm?

http://nerdhome.jimdo.com

Posted on
Tue Nov 21, 2017 8:00 am
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Re: Concurrent Thread Plugin level vs device level.

Argh. I thought about this overnight and on the drive in to work, and then typed the post and hit submit, and then the answer immediately came to me!

Does this make sense: I should define the variables in the devRunConcurrentThread method just as a variable, but all of my little sub-methods should never update the 'global' (or device-specific in this example) variables. They should just return values to wherever they were called from (devRunConcurrentThread), which can then assign them to the variable required.

Seems obvious now.

I think this just highlights my struggle to convert from 'scripting' to 'programming'. Thinking about multiple instances of everything and what exists where trips me up the most.

http://nerdhome.jimdo.com

Posted on
Tue Nov 21, 2017 10:12 am
FlyingDiver online
Posts: 1806
Joined: Jun 07, 2014
Location: Cape Coral, FL

Re: Concurrent Thread Plugin level vs device level.

Define a class that encapsulates all the methods and data for that device. Then the local variables (for a specific device) are just self.name within that class.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Tue Nov 21, 2017 6:12 pm
jay (support) offline
Site Admin
User avatar
Posts: 13363
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Concurrent Thread Plugin level vs device level.

FlyingDiver wrote:
Define a class that encapsulates all the methods and data for that device. Then the local variables (for a specific device) are just self.name within that class.


You could even make that class a subclass of the threading.Thread class so that it's all completely self contained. So, for each Indigo device, you create an instance of your subclass (pass in whatever data from the indigo device), then tell the instance to run. That would handle all your network communication, would do whatever transformations you need, etc.

Asynchronous programming is definitely somewhat challenging because it's so different in approach. But it can also make some things much easier once you get the hang of it.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Nov 28, 2017 1:20 pm
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Re: Concurrent Thread Plugin level vs device level.

I am plowing my way through this and making headway...

I've actually got this thing up and running (kind of like how a baby giraffe stands up and wobbles around). It seems to start up and shut down without much drama now, however I'm pretty sure it's not quite working correctly.

So as discussed, the plugin will have a network connection for each device (I'll only ever use one I'm sure... so in that sense it works, but I can easily see the use case for two or more devices).

Anyway, Once the device is running and reporting data I try starting up a second device (just entering bogus IP address, etc..). I have no delusions of getting anything on this second device (since I only have one to test) but as soon as I create it, I stop getting anything from the first device, and that shouldn't happen.

Clearly, I'm a bit out of my league here, but I think I know enough to realize that the two devices are probably sharing some space or scope, and that's where everything goes bad.

I'm at work, and don't have the code with me (but that's a good thing). I'll much more briefly outline what I have here in a sort of python markup, and hopefully somebody can tell me what the self I've done wrong.

Code: Select all
from threading import Thread
#So the heart is basically a procedure that will loop like the usual concurrent thread, but at the device level.

def runDevConcurrentThread(self, dev):
     While True:
          Try:
               data = self.socket.recv(1024)
               #then it does a bunch of stuff with the data it gets. All that works fine
               dev.updateStateOnServer("statename", datafromsocket)
          self.sleep(0.1)
         except: self.stopThread
               pass

#then the device start procedure starts this thread
def deviceStartComm(self, dev):
     #gets IP address, etc... from device properties, starts up the socket.
     #there's more lines to this, but ultimately it works and the socket gets data
     self.socket.bind((IP ADRR, PORT))
     #then start the thread. THIS WORKS, but it only took me a few tries, and so seems too easy...
     Thread(target=runDevConcurrentThread, args=(dev, ) ).run()

#And a method to kill this whole thing.
def deviceStopComm(self, dev):
     #First I close the socket, and then try to stop the thread. Seems to work, but as I type here, I think I should stop the thread first.
     self.socket.close()
     self.stopThread = True



That basically covers it. I guess I'm a little confused that the things I define in startComm and stopComm find their way to the right device. Well. Clearly they don't. self.stopThread probably just stops all of the threads, so it works just fine if you are shutting the plugin down (but really isn't the right way to do this. Same goes for the socket I'm guessing, and this time it causes problems. I think when I think I'm starting a second one, I'm probably just redefining the 'one' that I have and then nothing works.

For the most part I've been figuring this out with various internet sources, but the examples usually assume I know something about programming threads, etc.. and none of them are specific enough to what I'm doing here.

I figure I'll get this eventually, but if anyone has any insight and can save me a night or two it would be hugely appreciated.

http://nerdhome.jimdo.com

Posted on
Tue Nov 28, 2017 1:50 pm
FlyingDiver online
Posts: 1806
Joined: Jun 07, 2014
Location: Cape Coral, FL

Re: Concurrent Thread Plugin level vs device level.

You need to encapsulate the connection info for each device into a separate object. The easiest way to do that is to create a Python class representing the device, and instantiate (create) a class object for each device. Look at the sources for the BetterEmail plugin and look at how I do it for SMTP connections (for instance).

There are ways to do it without classes. You could create a Dict with the info, then you need to have a table (or List) of these dicts, each one representing a different connection.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Tue Nov 28, 2017 6:27 pm
jay (support) offline
Site Admin
User avatar
Posts: 13363
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Concurrent Thread Plugin level vs device level.

The problem is self.socket is a single socket connection - just one. So it works for the first device, but when you try to open the second one, who knows what's happening because you're trying to bind an already open socket. It probably closes the first connection and binds the second one but that's just a guess.

FlyingDiver is right: create a new class (outside of your Plugin subclass) that encapsulates the socket and communication, all the data, commands, etc. As I said above, I'd make it a subclass of Thread. Then, when one of your Indigo devices starts up, you can instantiate an object of your class type, pass in the data from the device (presumably the connection information) and probably a command Queue object such that when an action is executed, your action code puts the command on the queue. Then inside the run loop for your object instance you can continually pick commands off the queue and execute them.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Nov 30, 2017 8:24 am
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Re: Concurrent Thread Plugin level vs device level.

Well, after stopping my head from spinning with you guys telling me to create a new class, I dug in and I think this is going to work out very well. I seem to have had that 'moment' where I started to wrap my brain around the concept of a class (and as you would expect, the dreaded 'self' started to make a lot more sense with it!) Having individual, device specific sockets and referring to them makes sense to me now. It actually didn't take that much work to re-work this thing into that new paradigm.

It's not working YET though, and of course I have a few new questions.

My problem is how to refer to things in the 'Plugin' class inside MY class. For example, I'm trying to add some logging lines in, but the 'self' part of 'self.logger.debug' is going to refer to my class, not the Plugin class. Same goes for self.sleep. (I've mainly been shocked by how ineffective all of my googling has been here!)

I also have a question on exactly where, or on what level the class should be defined. Is it at the same level as the Plugin class, or indented so that it's 'within' the Plugin class? (And I suspect the answer to this question will affect the answer to the first question on how to reference the logger, etc...).

Thanks so much for the help so far - I feel like I'm learning a ton. FlyingDiver, I read through your BetterEmail plugin, and while it definitely added *some* confusion, it mostly helped me understand it and I definitely feel like it's brought me forward!

http://nerdhome.jimdo.com

Posted on
Thu Nov 30, 2017 11:33 am
jay (support) offline
Site Admin
User avatar
Posts: 13363
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Concurrent Thread Plugin level vs device level.

Swancoat wrote:
Well, after stopping my head from spinning with you guys telling me to create a new class, I dug in and I think this is going to work out very well. I seem to have had that 'moment' where I started to wrap my brain around the concept of a class (and as you would expect, the dreaded 'self' started to make a lot more sense with it!) Having individual, device specific sockets and referring to them makes sense to me now. It actually didn't take that much work to re-work this thing into that new paradigm.


Network programming is not easy, I applaud you for sticking with it.

Swancoat wrote:
My problem is how to refer to things in the 'Plugin' class inside MY class. For example, I'm trying to add some logging lines in, but the 'self' part of 'self.logger.debug' is going to refer to my class, not the Plugin class. Same goes for self.sleep. (I've mainly been shocked by how ineffective all of my googling has been here!)


The simple answer is to just pass the logger in when you create an instance of your class. In the __init__() method for your class for instance:

Code: Select all
def __init__(self, dev, logger):
   '''
   Init the instance with the following parameters

   :param dev: The Indigo device
   :param logger: The logger from the Plugin instance
   '''
   self.dev = dev
   self.logger = logger
   self.logger.threaddebug("Initing device...")


This can get more complex but provide more explicit logging by creating your own logger, but this is definitely the simplest solution. Note: you could pass in the entire Plugin instance instead so you have access to everything. I prefer being a bit more explicit about what's needed in the class, but either way works. Note: if your class is a threading.Thread subclass, there's more that you should consider.

Swancoat wrote:
I also have a question on exactly where, or on what level the class should be defined. Is it at the same level as the Plugin class, or indented so that it's 'within' the Plugin class? (And I suspect the answer to this question will affect the answer to the first question on how to reference the logger, etc...).


You can define it in either place or in fact in a completely different file. My preference is generally to define it in a different file if it's a meaty class (yours probably is) or at the top level (same as Plugin) in the plugin.py if the class is small and relatively single-purposed. My recommendation would be to add a new file (at the same level as plugin.py in the plugin bundle), let's say devices.py. Then, inside that file, you create the class:

Code: Select all
import socket
# All other imports

class XYZ(object):
    def __init__(self, dev, logger):
       '''
       Init the instance with the following parameters

       :param dev: The Indigo device
       :param logger: The logger from the Plugin instance
       '''
       self.dev = dev
       self.logger = logger
       self.logger.debug("Initing device...")

    # All the rest of your implementation here


Then, in plugin.py, you just:

Code: Select all
import devices


at the top and then later:

Code: Select all
dev startDeviceComm(self, device):
   # let's assume you created a dictionary in the Plugin.__init__() method that maps
   # a device ID to an instance of your device class called self.device_map
   try:
      new_device_instance = devices.XYZ(device, self.logger)
   except:
      # do whatever is appropriate for when a device can't be created and then
      # bail
      return
   # The new device instance is now created, so save it into the map so we can find
   # it later
   self.device_map[device.id] = new_device


This approach makes it easy for you to add other device classes in the devices.py file should that become necessary, and all the work is contained in that file rather than in a monolithic plugin.py.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Thu Nov 30, 2017 10:17 pm
Swancoat offline
Posts: 409
Joined: Nov 20, 2009
Location: Houston

Re: Concurrent Thread Plugin level vs device level.

This is outstanding Jay. Super clear - thank you!

http://nerdhome.jimdo.com

Who is online

Users browsing this forum: FlyingDiver and 2 guests