Multiprocessing and Indigo

Posted on
Wed Mar 29, 2017 11:07 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

jay (support) wrote:
Try explicitly stopping the Process instance (and it wouldn't hurt to make sure the Queue object is empty) before the script exits. The Python interpreter may think that those are still in scope (likely the Process object because it's managing a separate process at some level).

Thanks Jay.

No joy. I've tried explicitly stopping it and also running the process as a daemon with no luck. It's curious that I can't get it to complete successfully when embedded, but that's tangential to my main goal. I think I may be on the right path for that.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Mar 29, 2017 11:09 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

autolog wrote:
I have just run the above code as an embedded script and it is working instantly. :)

That is weird.

I've tried many iterations and can't get the script to run as embedded. When I paste it into the Indigo Scripting Shell, it also runs properly.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Mar 29, 2017 11:12 am
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

autolog wrote:
Hi Dave,
I think you are on the right track the design could be something like:

You could have a thread running that receives requests from an input queue to create/update charts.
This thread invokes the multiprocess process to call matplotlib. It doesn't need to do a join as you are just launching it and expect/hope it to work.
It updates the chart device status to say "Charting requested"

The matplotlib process adds its result to an output queue

You have another thread that waits on these output messages from matplotlib.
It updates the chart device status to say "Charting completed" and/or error intercepted.

In the event that matplotlib crashes and no message is queued then you could have a timer set to flag this.

In this way you are isolating matplotlib from the plugin processes. :)

Thanks Jon - it sounds like we're on the same wavelength. Now I just need to make it happen. :D

Appreciate the help!

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Mar 29, 2017 4:16 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

Following up on the plugin part of the (ahem) process, multiprocessing appears to work well from within a plugin also. This construction seems to work well:

Code: Select all
from multiprocessing import Process, ProcessError, Queue

def method_name(queue, arg=None):

    arg += 1
    queue.put(arg)

if __name__ == '__main__':
    queue = Queue()
    p1 = Process(name='Foo', target=self.method_name, args=(queue, 1))
    p1.start()
    p1.join()

    result = queue.get()
    indigo.server.log(result)
    queue.put('')

It may not matter much, but for my test, I ran the main part of the code from within runConcurrentThread.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Mar 29, 2017 4:24 pm
jay (support) offline
Site Admin
User avatar
Posts: 18200
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Multiprocessing and Indigo

Does the plugin shut down properly or is it killed by Indigo after some period of time when you try to disable it? Just curious.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Wed Mar 29, 2017 7:21 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

jay (support) wrote:
Does the plugin shut down properly or is it killed by Indigo after some period of time when you try to disable it? Just curious.

The plugin shut down quite calmly. :D

Just for poops and giggles I tried accessing the indigo.* API from within the secondary process method call (in the plugin) and indigo.server.log() was not honored as expected. Since the queue will accept lots of different data structures like tuples, lists and dicts, this will be easy to handle.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri Apr 07, 2017 1:44 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

For posterity. Both scripts have been tested against Indigo 7.

Here is a simple plugin.py file that seems to run very stably in Indigo (although it shows the plotting method in a different class, it doesn't need to be):

Code: Select all
#! /usr/bin/env python2.6
# -*- coding: utf-8 -*-

import multiprocessing as mp
import matplotlib
matplotlib.use('AGG')
import numpy.random as random
from os.path import expanduser

class matplot_stuff():

    def do_plot(self, returnQueue):

        import matplotlib.pyplot as plt

        try:
           
            a = random.sample(500)
            b = random.sample(500)

            fig = plt.figure(1)
            plt.scatter(a, b, color='red', edgecolor='black')
            plt.savefig('{0}/Desktop/test_plot.png'.format(expanduser("~")))
            plt.close()
            plt.clf()

            returnQueue.put({'Message': 'Success', 'Error': False})
           
        except Exception, e:
            returnQueue.put({'Message': unicode(e), 'Error': True})


class Plugin(indigo.PluginBase):

    def __init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs):
        indigo.PluginBase.__init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs)

    def runConcurrentThread(self):

        try:
            returnQueue = mp.Queue()
            multis = matplot_stuff()
           
            while True:
                if __name__ == '__main__':
                    p1 = mp.Process(name='p1', target=multis.do_plot, args=(returnQueue,))
                    p1.start()
                    p1.join()

                result = returnQueue.get()
               
                if result['Error']:
                    indigo.server.log(unicode(result['Message']), isError=True)
                else:
                    indigo.server.log(unicode(result['Message']))

                self.sleep(60)

        except StandardError as e:
            indigo.server.log("Error in runConcurrentThread: {0}".format(e))


Result:
Code: Select all
Apr 7, 2017, 2:40:27 PM
   Multiprocess                    Success


Here is a simpler demonstration using multiprocess pools.

Code: Select all
#! /usr/bin/env python2.7
# -*- coding: utf-8 -*-

import multiprocessing
import matplotlib
matplotlib.use('AGG')
import matplotlib.pyplot as plt
import numpy.random as random
from os.path import expanduser


def mpl_scatter((name, obs, obs_color)):

    random.seed() 
    a = random.sample(obs)
    b = random.sample(obs)

    plt.scatter(a, b, color=obs_color)
    plt.savefig('{0}/Desktop/test_scatter_{1}.png'.format(expanduser("~"), obs_color))
    plt.close()
    plt.clf()

    return '{0} done'.format(name)

class Plugin(indigo.PluginBase):

    def __init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs):
        indigo.PluginBase.__init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs)

    def runConcurrentThread(self):

        try:
            while True:
           
                if __name__ == "__main__":
                    data = (['a', 250, 'red'], ['b', 500, 'blue'], ['c', 1000, 'green'],)
                    p = multiprocessing.Pool() 

                    result = p.map(mpl_scatter, data)  # Returns a list
                    indigo.server.log(unicode(result))

                self.sleep(60)

        except StandardError as e:
            indigo.server.log("Error in runConcurrentThread: {0}".format(e))


Result:

Code: Select all
Apr 7, 2017, 2:39:36 PM
   Multiprocess                    ['a done', 'b done', 'c done']

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Nov 14, 2020 6:54 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

A word of caution when using multiprocessing within a plugin. With the amazing help of Matt & Jay, we've found that there's a potential pitfall to this approach if you send elements to a child process that rely on calls to the Indigo API (arguments sent to a multiprocessing process need to be pickled and Indigo API objects are not pickleable).

Sending Indigo device objects or instances of the host plugin will not work well--they may look like they're working (and aspects of them may work)--but they're actually being forked and not being implemented properly by the multiprocessing library. For example, calls to dev.pluginProps() may work, but calls to the host plugin logging process won't work at all. So, instead of sending an indigo.Dict() to a process -- send a standard Python dict instead. Instead of making calls to self.logger() -- you need to establish a separate logging process (sending data to the Indigo event log will require a Queue).

Matt & Jay, please feel free to correct anything I've mischaracterized.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Nov 15, 2020 11:25 am
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Multiprocessing and Indigo

Thanks for posting the details Dave. Looks accurate to me.

Image

Posted on
Sun Nov 15, 2020 12:12 pm
FlyingDiver offline
User avatar
Posts: 7189
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Multiprocessing and Indigo

This is specific to multiprocessing, not multithreading within one process, correct?

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Nov 15, 2020 12:24 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

Hmm. I would say that this is with respect to pickling Indigo objects. If the arguments sent to the thread are pickled, you might want to try to find a safer structure. One of the most troubling things we found is the operation could fail silently.

In my case, I was sending the Indigo objects to each process which in turn loaded a separate class (separate from the plugin class). I also sent an instance of the plugin class (likewise improperly pickled) and it was remarkable how much worked in spite of the limitations.

If you do a test within the plugin class and try to pickle an Indigo object, it should fail instantly.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Nov 15, 2020 12:30 pm
FlyingDiver offline
User avatar
Posts: 7189
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Multiprocessing and Indigo

Well, I don't use the pickle module anywhere, so I expect this isn't an issue. I'll keep it in mind if I should ever want to do that.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Sun Nov 15, 2020 12:58 pm
DaveL17 offline
User avatar
Posts: 6744
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Multiprocessing and Indigo

In this case, it's apparently the multiprocessing library that's doing the pickling, so you might want to check if anything downstream is doing it. I suspect not, but you never know.

This all came up as I'm chasing a very strange bug. The cause of the bug still remains a mystery. One fun bit of this exercise is when I turn on PyCharm debugging, the bug literally doesn't happen.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Nov 15, 2020 1:37 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Multiprocessing and Indigo

I think threading should be fine. The underlying issue appears to be that when the multiprocessing module spawns a new process (by forking the current one) it creates a crippled version of the Indigo Plugin Host (which is what defines all of the Indigo APIs that are accessible in python). Some of them work, but the ones that have to communicate back to the Indigo Server definitely fail.

Image

Posted on
Sun Nov 15, 2020 1:39 pm
FlyingDiver offline
User avatar
Posts: 7189
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Multiprocessing and Indigo

Thanks Matt. I've never used the multiprocessing module either. Just lots of threads.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Who is online

Users browsing this forum: No registered users and 2 guests