Stuck moving code to runConcurrentThread()

Posted on
Fri Jul 31, 2020 2:22 pm
hannt offline
User avatar
Posts: 84
Joined: Jul 08, 2011
Location: TN USA

Stuck moving code to runConcurrentThread()

I've created a module with a single class that creates a graph using matplotlib. This is how I initialize the class and create the graph:

Code: Select all
   def createGraphFromDev(self, devId):
      theGraph = stategrapher.DateValueLineGraph(devId, self.pluginPrefs['DatabaseFile'], self.logger)
      theGraph.create_graph()
The dev (devId) contains in its properties all of the information needed to create the graph.

Code: Select all
import matplotlib.pyplot as plt

class DateValueLineGraph():
   ########################################
   def __init__(self, devId, database_file, logger):
      self.logger = logger
      self._dev = indigo.devices[int(devId)]
      self._database_file = database_file
      self._fig, self._ax = plt.subplots()
      ...
This all works fine, however, it's a little slow. It makes several queries to an sqlite database to get the data then creates the graph. I'm trying to move the graph creation into runConcurrentThread() so nothing is blocked while the graph is created. So I setup a queue and put the job in the queue:

Code: Select all
   def createGraphFromDev(self, devId):
      self.jobQueue.put(devId)
Then I moved the two lines of code to runConcurrentThread() to create the graph:

Code: Select all
def runConcurrentThread(self):
      try:
         while True:
            if not self.jobQueue.empty():
               devId = self.jobQueue.get()
               theGraph = stategrapher.DateValueLineGraph(devId, self.pluginPrefs['DatabaseFile'], self.logger)
               theGraph.create_graph()
            self.sleep(1)
      except self.StopThread:
         self.logger.debug(u"runConcurrentThread() exception StopThread")
When I try to create the graph, I get this error:

Code: Select all
State Grapher Error             Error in plugin execution runConcurrentThread
State Grapher Error             plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)
Since there was no helpful error message, I tried reloading the plugin in the Interactive Shell. When run in the interactive shell, the line of code that causes an error is a simple import:

Code: Select all
import matplotlib.pyplot as plt
In the interactive shell, this message is displayed when the code gets to the import statement:

Code: Select all
Connected to Indigo Server v7.4.1, api v2.3 (localhost:1176)
Started Plugin State Grapher v0.0.390
>>> 2020-07-31 14:37:52.164 IndigoPluginHost[6915:97316] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'nextEventMatchingMask should only be called from the Main Thread!'
*** First throw call stack:
(
0   CoreFoundation                      0x00007fff35746be7 __exceptionPreprocess + 250
1   libobjc.A.dylib                     0x00007fff6e3855bf objc_exception_throw + 48
2   AppKit                              0x00007fff3293eb65 +[NSEvent _discardTrackingAndCursorEventsIfNeeded] + 0
3   _macosx.so                          0x0000000107e16a8d init_macosx + 1206
4   readline.so                         0x0000000106b5ac8e initreadline + 950
5   Python                              0x00007fff408370b2 PyOS_Readline + 179
6   Python                              0x00007fff408b21f2 _PyBuiltin_Init + 12167
7   Python                              0x00007fff408b8bbf PyEval_EvalFrameEx + 18888
8   Python                              0x00007fff408b3bec PyEval_EvalCodeEx + 531
9   Python                              0x00007fff408bd6ea _PyEval_SliceIndexNotNone + 459
10  Python                              0x00007fff408b8b00 PyEval_EvalFrameEx + 18697
11  Python                              0x00007fff408b3bec PyEval_EvalCodeEx + 531
12  Python                              0x00007fff4085aaa8 PyFunction_SetClosure + 772
13  Python                              0x00007fff4083d143 PyObject_Call + 97
14  _functools.so                       0x0000000106a297ee init_functools + 470
15  Python                              0x00007fff4083d143 PyObject_Call + 97
16  Python                              0x00007fff408b9276 PyEval_EvalFrameEx + 20607
17  Python                              0x00007fff408bd68f _PyEval_SliceIndexNotNone + 368
18  Python                              0x00007fff408b8b00 PyEval_EvalFrameEx + 18697
19  Python                              0x00007fff408bd68f _PyEval_SliceIndexNotNone + 368
20  Python                              0x00007fff408b8b00 PyEval_EvalFrameEx + 18697
21  Python                              0x00007fff408b3bec PyEval_EvalCodeEx + 531
22  Python                              0x00007fff4085aaa8 PyFunction_SetClosure + 772
23  Python                              0x00007fff4083d143 PyObject_Call + 97
24  Python                              0x00007fff408478f4 PyMethod_New + 1169
25  Python                              0x00007fff4083d143 PyObject_Call + 97
26  Python                              0x00007fff408bcfeb PyEval_CallObjectWithKeywords + 159
27  Python                              0x00007fff408e940b initthread + 2815
28  Python                              0x00007fff408e5103 PyThread_start_new_thread + 279
29  libsystem_pthread.dylib             0x00007fff6f731109 _pthread_start + 148
30  libsystem_pthread.dylib             0x00007fff6f72cb8b thread_start + 15
)
libc++abi.dylib: terminating with uncaught exception of type NSException
/Library/Application Support/Perceptive Automation/Indigo 7.4/.debugPluginLaunch.command: line 7:  6915 Abort trap: 6           "/Library/Application Support/Perceptive Automation/Indigo 7.4/IndigoPluginHost.app/Contents/MacOS/IndigoPluginHost" -d200 -p1176 -f"State Grapher.indigoPlugin"
Here's some additional information. Just for testing purposes, I commented out the line of code that imports the module for graphing and I commented out the code that calls it. I then moved the include statement to plugin.py (instead of in the module). Even running this in the interactive shell causes the same error. This leads me to believe that there are two different problems going on here.

  1. For whatever reason, you can't run the plugin in the interactive shell if it includes matplotlib.pyplot.
  2. The other problem, which may or may not be related, is that the code still errors out when it is not run in the interactive shell. In this scenario, the code does not have a problem with the include statement, but errors out when trying to initialize self._fig and self._ax:

Code: Select all
import matplotlib.pyplot as plt

class DateValueLineGraph():
   ########################################
   def __init__(self, devId, database_file, logger):
      self.logger = logger
      self._dev = indigo.devices[int(devId)]
      self._database_file = database_file
      self._fig, self._ax = plt.subplots()
      ...
Any thoughts? I don't even know where to begin with this one.

Posted on
Fri Jul 31, 2020 2:41 pm
FlyingDiver offline
User avatar
Posts: 7189
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Stuck moving code to runConcurrentThread()

You need to put the graph generation in it's own thread, not in the plugin's existing threads.

In my Lutron plugin, when I do the automatic device creation, I do:

Code: Select all
        deviceThread = threading.Thread(target = self.createCasetaDevices, args = (valuesDict, ))
        deviceThread.start()   


So you should probably do something like that, then the last thing the routine should so is set a flag that it's done, and your runConcurrentThread should check for that flag.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Fri Jul 31, 2020 6:30 pm
hannt offline
User avatar
Posts: 84
Joined: Jul 08, 2011
Location: TN USA

Re: Stuck moving code to runConcurrentThread()

FlyingDiver wrote:
You need to put the graph generation in it's own thread, not in the plugin's existing threads.

Thanks for the response, Joe. I just tried putting the graph generation in a separate thread as you suggested and had similar results. Everything is pointing to matplotlib.pyplot.subplots() as the culprit. That call is where the code quits when in a separate thread, which is the same as when it is in runConcurrentThread.

I'll keep pecking at this in hopes of narrowing down the issue. The challenge though is that I can't see errors from the thread unless I restart the plugin in the interactive shell. But I can't start the plugin in the interactive shell because it errors out on the import of matplotlib.pyplot. :? I haven't done much with threads so maybe there's a better way to see errors than the interactive shell.

Posted on
Fri Jul 31, 2020 6:49 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Stuck moving code to runConcurrentThread()

Also matplot has a memory leak.
It will eat up memory over time.
I have put the matplot into an external py program that restarts itself every 100 iterations . It reads the data dict from a file

Although poorly documented you can look at indigoplotd plugin
It reads from sql ( SQLite or psql) and from device states, makes a flat file and that is used to plot w either matplot or gnuplot to generate the plots.

It is 5 years old - one of my first plugins.
.. still working.


Sent from my iPhone using Tapatalk

Posted on
Sat Aug 01, 2020 6:16 am
hannt offline
User avatar
Posts: 84
Joined: Jul 08, 2011
Location: TN USA

Re: Stuck moving code to runConcurrentThread()

kw123 wrote:
Also matplot has a memory leak.
It will eat up memory over time.

That probably explains the weirdness I've been experiencing with my Indigo server computer. At times it may take 10 seconds or more just to open a Finder window. Currently, I'm using a schedule to generate 2 graphs every 5 minutes using Matplotlib (it used to be every 2 minutes). Think I'll tinker around with disabling this schedule to see how it affects performance. The plugin I'm working on will eventually replace this Schedule/Action Group.

Anyway, back to the issue. I think the answer to the problem I'm running into can be found in the threading section of the Matplotlib documentation. It seems that in general, Matplotlib is not thread safe and wants to be run on the main thread. All errors I have captured seem to point to creating the plot on the main thread.

kw123 wrote:
I have put the matplot into an external py program that restarts itself every 100 iterations . It reads the data dict from a file.

This sounds like a strategy I need to look at in more detail.

Thanks for the information, Karl.

Posted on
Sat Aug 01, 2020 8:34 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Stuck moving code to runConcurrentThread()

The matplot py program in the plugin Dir is using a function to measure the memory consumption.


Sent from my iPhone using Tapatalk
[edit]
i used:
import getrusage
resource.getrusage(resource.RUSAGE_SELF)

that delivers a number of measurement of allocated resources to the ppm

Posted on
Sat Aug 01, 2020 3:47 pm
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Stuck moving code to runConcurrentThread()

With the Matplotlib plugin, I found the combination of three things has been able to manage the memory leak issue:

  1. a "special" extra import:
    Code: Select all
    import matplotlib
    matplotlib.use('AGG')  # Note: this statement must be run before any other matplotlib imports are done.
    import matplotlib.pyplot as plt
  2. Doubling-down on garbage collection:
    Code: Select all
        plt.clf()
        plt.close('all')
  3. Running each plotting operation in its own queue/thread.
Supposedly, later versions of Matplotlib have fixed the leak, but I've stuck with 1.3.1 which is what OS X ships with.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Aug 01, 2020 3:49 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Stuck moving code to runConcurrentThread()

hannt wrote:
Anyway, back to the issue. I think the answer to the problem I'm running into can be found in the threading section of the Matplotlib documentation. It seems that in general, Matplotlib is not thread safe and wants to be run on the main thread. All errors I have captured seem to point to creating the plot on the main thread.

Yeah, glancing at the Matplotlib documentation it looks like it is wanting to wait for keyboard input, which coincides with the initreadline() / PyOS_Readline() functions shown in the crash call stack. I'm not sure of the solution but it doesn't surprise me if calling into those methods from a secondary thread, especially from the an IndigoPluginHost (versus a normal interactive python shell), is going to crash. It sounds like Karl got it to work by running matplot in its own process.

Image

Posted on
Sat Aug 01, 2020 4:13 pm
hannt offline
User avatar
Posts: 84
Joined: Jul 08, 2011
Location: TN USA

Re: Stuck moving code to runConcurrentThread()

DaveL17 wrote:
With the Matplotlib plugin, I found the combination of three things has been able to manage the memory leak issue:

  1. a "special" extra import:
    Code: Select all
    import matplotlib
    matplotlib.use('AGG')  # Note: this statement must be run before any other matplotlib imports are done.
    import matplotlib.pyplot as plt
  2. Doubling-down on garbage collection:
    Code: Select all
        plt.clf()
        plt.close('all')
  3. Running each plotting operation in its own queue/thread.
Supposedly, later versions of Matplotlib have fixed the leak, but I've stuck with 1.3.1 which is what OS X ships with.

Wow! matplotlib.use('AGG') seems to have solved the original problem. Thank you! It even works in runConcurrentThread, but from this discussion, it sounds like a better strategy to run it from it's own thread as I also want to stick with v1.3.1.

Posted on
Sat Aug 01, 2020 4:55 pm
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Stuck moving code to runConcurrentThread()

Good deal. I was chasing my tail for a long time with that dang leak myself.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Aug 01, 2020 5:32 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Stuck moving code to runConcurrentThread()

... and Matplot is a resource hog. It uses a factor of 50 more cpu than gnuplot for each plot.


Sent from my iPhone using Tapatalk

Posted on
Sat Aug 01, 2020 6:27 pm
hannt offline
User avatar
Posts: 84
Joined: Jul 08, 2011
Location: TN USA

Re: Stuck moving code to runConcurrentThread()

Going back to the thread topic...

FlyingDiver wrote:
You need to put the graph generation in it's own thread, not in the plugin's existing threads.

DaveL17 wrote:
Running each plotting operation in its own queue/thread.[/list]
Supposedly, later versions of Matplotlib have fixed the leak, but I've stuck with 1.3.1 which is what OS X ships with.

Creating a graph that is saved to a file is now working fine in runConcurrentThread. Additionally, that would be the only purpose in my plugin for using runConcurrentThread. What is the reasoning for using a separate thread? (Just trying to understand) Thanks!

Posted on
Sat Aug 01, 2020 7:00 pm
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Stuck moving code to runConcurrentThread()

From my perspective, first and foremost, it's the ability to destroy it when it's finished.

Indigo is running your plugin in a thread that it controls. This is so my code doesn't take down your code (and for security, etc.) When things go really wrong, Indigo can try to kill and spawn my plugin anew. Sometimes, the trouble is so severe that it can't do this successfully, and this helps keep the rest of the system safe.

runConcurrentThread() is optional. You don't have to have one.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Aug 02, 2020 6:23 am
hannt offline
User avatar
Posts: 84
Joined: Jul 08, 2011
Location: TN USA

Re: Stuck moving code to runConcurrentThread()

DaveL17 wrote:
From my perspective, first and foremost, it's the ability to destroy it when it's finished.

That makes a lot of sense. Thanks.

I keep going back to this thread where Jay suggests runConcurrentThread() instead of creating a separate thread:

jay (support) wrote:
If so, I'd recommend taking @howartp's recommendation: create a Python Queue when the plugin starts up. When the action is called, add something to the queue. In runConcurrentThread, periodically check the queue to see if there's something to do, and if so then pull it off the queue and process it. Much easier than starting/stopping threads, and having runConcurrentThread run in a relatively tight loop isn't a resource contention.

Assuming the task to be run on a separate thread isn't too complex, I can see where either strategy would work just fine.

Posted on
Sun Aug 02, 2020 6:59 am
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Stuck moving code to runConcurrentThread()

Yeah, the idea of starting and killing threads in this instance is solely a measure to overcome the memory leak in Matplotlib. During development, I tried a ton of alternatives and what I settled on seemed to be the LOA (least objectionable alternative). :D

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Who is online

Users browsing this forum: No registered users and 1 guest