Multiprocessing and Indigo

[DaveL17]

I'm starting to experiment with multiprocessing in one of my plugins, but I'm getting results I don't understand. If I try to run a separate process in an embedded or linked script, I don't seem to get any results. However, while it *appears* that I'm able to start a separate process within my own plugin, I can't seem to figure out a way to successfully name the process or successfully pass any arguments to the method being called by the process. A simple example:

Code: Select all

import multiprocessing

def method_name(self, arg=None):
    indigo.server.log(str(arg))
    indigo.server.log(multiprocessing.current_process().name)

p1 = multiprocessing.Process(name='Foo', target=self.method_name(), args=(arg_to_pass,))
p1.start()
p1.join()

The above seems to run successfully when I invoke it within my plugin, but when I try to access the args that I passed, I get the NoneType default and I don't get the process name 'Foo' but rather the name 'MainProcess'.

Can someone smarter than me please provide an example of how to get a separate process to run within a plugin? I clearly have more to figure out.

[bkmar1192]

I use multi-threading in my Security Camera plugin. Feel free to check it out and ping with questions.


Sent from my iPhone using Tapatalk

[DaveL17]

Thanks for the reply @bkmar1192, but I'm interested specifically in the mulitprocessing module (not the threading module). My research says that using multiple processes would be more stable than using threads for the thing that I'm trying to accomplish. I may have to fall back on threading if it turns out that multiprocessing is incompatible with the Indigo host process.

Thanks again.

[jay (support)]

Yeah, Python's threads are less than optimal because of the GIL, particularly for processor intensive things. I've never used the multiprocessing module, much less trying to use it from a plugin. I suspect (but it's only a guess) that because of the way we run plugins it might not work right/well but I don't know for sure. Matt might know more.

[DaveL17]

Yeah, Python's threads are less than optimal because of the GIL, particularly for processor intensive things. I've never used the multiprocessing module, much less trying to use it from a plugin. I suspect (but it's only a guess) that because of the way we run plugins it might not work right/well but I don't know for sure. Matt might know more.

Thanks Jay. I've moved my work to threads to see how well that will manage my need, but would still prefer to use multiple processes if that's possible (interestingly, threads do seem to be working okay so far). When I looked into threads, I found that there are situations where a problem in one thread can bring down other threads. Assuming that I'm already in a thread (runConcurrentThread), that news concerned me. It sounds like multiprocessing doesn't suffer from that malady.

Hopefully Matt will have a chance to chime in with the ingredients to the secret sauce.

[RogueProeliator]

When I looked into threads, I found that there are situations where a problem in one thread can bring down other threads.

I am far from sure on this, but wouldn't the primary cause of this be unhandled exceptions? I would think that if all the work in the thread is properly error trapped, it would not be too likely to have a situation where one thread kills another... ? I use threads extensively in my plugins and have not had any issues like that (many of my plugins create one thread per device, at least for A/V ones that should remain responsive if two are being used at the same time).

Adam

[RogueProeliator]

Also, just a warning, but generally speaking inter-process communications are generally far more difficult than communications between threads within the same process. I've never tried this in Python, just speaking generally from other languages... so if your use case involves needing to communicate between the plugin and the worker thread/process you may want to investigate that aspect.

[matt (support)]

I'm not sure how well the python multiprocessing module will work from the context of an Indigo script (or plugin), but I am curious to know if it works. I think the main problem you are having is just a syntax error on the target argument. I believe this line:

Code: Select all

p1 = multiprocessing.Process(name='Foo', target=self.method_name(), args=(arg_to_pass,))

should be:

Code: Select all

p1 = multiprocessing.Process(name='Foo', target=self.method_name, args=(arg_to_pass,))

To avoid having the method self.method_name called before Process() is invoked.

I think trying to call indigo.* APIs from the secondary process is going to fail. I don't think it will have access to the indigo.* module / namespace since it is special and provided by the custom python runtime under which we execute scripts (and plugins). You might be able to use a Queue though to pass messages from the secondary process to the main process. So in this example instead of having the secondary process directly call indigo.server.log() you could pass the log messages to log message queue (which you add) that the main process handles by calling indigo.server.log.

[DaveL17]

When I looked into threads, I found that there are situations where a problem in one thread can bring down other threads.

I am far from sure on this, but wouldn't the primary cause of this be unhandled exceptions? I would think that if all the work in the thread is properly error trapped, it would not be too likely to have a situation where one thread kills another... ? I use threads extensively in my plugins and have not had any issues like that (many of my plugins create one thread per device, at least for A/V ones that should remain responsive if two are being used at the same time).

Adam

This effort is to see if multiprocessing can be used as a trap for the matplotlib plugin restarts which are caused by a memory leak in matplotlib. Eventually, the leak brings the whole plugin down and I'm trying to see if I might be able to farm select things out to there own processes so that if a single process fails the parent will remain alive. I am also far from being an expert in this, but if one thread can bring down another thread, then using threads may not be sufficient for this particular nut.

At least that's what I'm telling myself.

[DaveL17]

Good catch on the typo Matt - I completely missed that.

I think trying to call indigo.* APIs from the secondary process is going to fail. I don't think it will have access to the indigo.* module / namespace since it is special and provided by the custom python runtime under which we execute scripts (and plugins). You might be able to use a Queue though to pass messages from the secondary process to the main process. So in this example instead of having the secondary process directly call indigo.server.log() you could pass the log messages to log message queue (which you add) that the main process handles by calling indigo.server.log.

This exceptionally simple example works in Indigo:

Code: Select all

from multiprocessing import Process, Queue

def method_name(queue, arg=None):

    arg += 1
    queue.put(arg)
    
queue = Queue()
p1 = Process(name='Foo', target=method_name, args=(queue, 1))
p1.start()
p1.join()

result = queue.get()
indigo.server.log(str(result))

If run as an embedded script, it will cause the executor to timeout, which is odd because it only takes a second to run when run as a linked script (I've yet to try it as a plugin.) I could never get a call to the Indigo.* API to work from within the secondary process. If I try to import Indigo into the secondary process, the entire thing fails. Otherwise, the script runs quickly and returns the expected result. So I suspect (but don't know for sure) that anything Indigo-y will need to be done in the main process.

[kw123]

Dave. I gave up on matplotlib running in indigo early on due to the memory issue and python 2.7 not running under indigo in v6. Mine runs completely outside and restarts itself after xx rounds or if memory usage goes up > 200 Mbyte.
Then the communication is done though a file with Json data. That seems to work fine. The delay in commands is 1 second ( the matplotlib Program checks a command file every second.



Sent from my iPhone using Tapatalk

[jay (support)]

If run as an embedded script, it will cause the executor to timeout, which is odd because it only takes a second to run when run as a linked script (I've yet to try it as a plugin.)

Try explicitly stopping the Process instance (and it wouldn't hurt to make sure the Queue object is empty) before the script exits. The Python interpreter may think that those are still in scope (likely the Process object because it's managing a separate process at some level).

I could never get a call to the Indigo.* API to work from within the secondary process. If I try to import Indigo into the secondary process, the entire thing fails. Otherwise, the script runs quickly and returns the expected result. So I suspect (but don't know for sure) that anything Indigo-y will need to be done in the main process.

Yep, that's what I would expect. The IPH process does some trickery to make the indigo module work correctly and since the Process object is starting a standard Python interpreter it's not available.

[autolog]

Good catch on the typo Matt - I completely missed that.

I think trying to call indigo.* APIs from the secondary process is going to fail. I don't think it will have access to the indigo.* module / namespace since it is special and provided by the custom python runtime under which we execute scripts (and plugins). You might be able to use a Queue though to pass messages from the secondary process to the main process. So in this example instead of having the secondary process directly call indigo.server.log() you could pass the log messages to log message queue (which you add) that the main process handles by calling indigo.server.log.

This exceptionally simple example works in Indigo:

Code: Select all

from multiprocessing import Process, Queue

def method_name(queue, arg=None):

    arg += 1
    queue.put(arg)
    
queue = Queue()
p1 = Process(name='Foo', target=method_name, args=(queue, 1))
p1.start()
p1.join()

result = queue.get()
indigo.server.log(str(result))

If run as an embedded script, it will cause the executor to timeout, which is odd because it only takes a second to run when run as a linked script (I've yet to try it as a plugin.) I could never get a call to the Indigo.* API to work from within the secondary process. If I try to import Indigo into the secondary process, the entire thing fails. Otherwise, the script runs quickly and returns the expected result. So I suspect (but don't know for sure) that anything Indigo-y will need to be done in the main process.

Looks like you are making good progress.

I have just run the above code as an embedded script and it is working instantly.

[autolog]

Hi Dave,
I think you are on the right track the design could be something like:

You could have a thread running that receives requests from an input queue to create/update charts.
This thread invokes the multiprocess process to call matplotlib. It doesn't need to do a join as you are just launching it and expect/hope it to work.
It updates the chart device status to say "Charting requested"

The matplotlib process adds its result to an output queue

You have another thread that waits on these output messages from matplotlib.
It updates the chart device status to say "Charting completed" and/or error intercepted.

In the event that matplotlib crashes and no message is queued then you could have a timer set to flag this.

In this way you are isolating matplotlib from the plugin processes.

[DaveL17]

Dave. I gave up on matplotlib running in indigo early on due to the memory issue and python 2.7 not running under indigo in v6. Mine runs completely outside and restarts itself after xx rounds or if memory usage goes up > 200 Mbyte.
Then the communication is done though a file with Json data. That seems to work fine. The delay in commands is 1 second ( the matplotlib Program checks a command file every second

Hi Karl - thanks. You had mentioned that on another thread as well. The reason I'd rather not take that approach is--if I'm thinking about it right--it would require a wholesale refactoring of the plugin. If I can get multiprocessing to work as intended, it will be a much easier approach with largely the same outcome.

That's the plan, anyway.