Python x2 and using more CPU?

Posted on
Fri Nov 25, 2016 3:18 pm
Different Computers offline
User avatar
Posts: 2533
Joined: Jan 02, 2016
Location: East Coast

Python x2 and using more CPU?

I started looking at this because my iTunes server on my Indigo machine suddenly started dropping signal/temporarily freezing. It's not just airplay--it stops locally too.

Looking at Activity Monitor, for the first time ever I see Python listed as a process--certainly not because it never ran before, but before I doubt it ever reached 1% utilization. Now I see one instance of Python taking anywhere from 0.5 to 70% of CPU time, and that latter often if not exclusively corresponds to iTunes hiccuping.

But there's also ANOTHER instance of Python in Activity Monitor, one that never seems to go above 1%

This happens even after a fresh restart of the whole computer.

Perhaps unrelated, but it also seems that various IndigoPluginServer processes, particularly EPS Device Extension and Hue Lights, are taking more processing time than they used to.
Open files for the greedy Python:
Code: Select all
/Library/Application Support/Perceptive Automation/Indigo 7/Plugins/INDIGOplotD.indigoPlugin/Contents/Server Plugin
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
/System/Library/Frameworks/Python.framework/Versions/2.7/Python
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/_dotblas.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/lib/_compiled_base.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/future_builtins.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/multiarray.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/umath.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/scalarmath.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/linalg/lapack_lite.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/linalg/_umath_linalg.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/fft/fftpack_lite.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/random/mtrand.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/_path.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_csv.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/ft2font.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/unicodedata.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/_png.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/_cntr.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/_image.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/_delaunay.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/_tri.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/backends/_backend_agg.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/special/_ufuncs.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/special/_ufuncs_cxx.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/special/specfun.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/interpolate/_fitpack.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/interpolate/dfitpack.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/linalg/_fblas.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/linalg/_flinalg.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/linalg/_flapack.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/linalg/calc_lwork.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/interpolate/interpnd.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_csr.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_csc.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_coo.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_dia.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_bsr.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_csgraph.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/csgraph/_shortest_path.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/csgraph/_tools.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/csgraph/_traversal.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/csgraph/_min_spanning_tree.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/spatial/ckdtree.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/spatial/qhull.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/spatial/_distance_wrap.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/resource.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/mpl-data/fonts/ttf/Vera.ttf
/usr/lib/dyld
/private/var/db/dyld/dyld_shared_cache_x86_64
/dev/null
/Users/macadmin/Documents/INDIGOplotD/matplot/matplot.log
/Users/macadmin/Documents/INDIGOplotD/matplot/matplot.log
/Users/macadmin/Documents/INDIGOplotD/matplot/matplot.log
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/SystemAppearance.car
/Library/Application Support/Perceptive Automation/Indigo 7/Logs/com.karlwachs.INDIGOplotD/plugin.log
->0x207ab4aae14f9f23
->0x207ab4aae14fa163
/dev/urandom

And for the non-greedy one:
Code: Select all
/Library/Application Support/Perceptive Automation/Indigo 7/IndigoWebServer
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
/System/Library/Frameworks/Python.framework/Versions/2.7/Python
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/objc/_objc.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/parser.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/CoreFoundation/_inlines.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/CoreFoundation/_CoreFoundation.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Foundation/_inlines.so
/usr/lib/libexpat.1.dylib
/usr/share/icu/icudt55l.dat
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Foundation/_Foundation.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/WebKit/_WebKit.so
/Library/Application Support/Perceptive Automation/Indigo 7/IndigoWebServer/Cheetah/_namemapper.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/OpenSSL/crypto.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/OpenSSL/rand.so
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/OpenSSL/SSL.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/unicodedata.so
/usr/lib/dyld
/private/var/db/dyld/dyld_shared_cache_x86_64
/dev/null
/dev/null
/dev/null
*:8176
unknown.domain:8176->old-ipad-i.domain:54444
localhost:49483->localhost:indigo-server
unknown.domain:8176->med-ed-ipad-3.domain:59612
/dev/urandom
->0x207ab4aae0e78993
count=2, state=0x12

Ideas? Is matplotlib really the culprit?

SmartThings refugee, so happy to be on Indigo. Monterey on a base M1 Mini w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKitLink, Grafana, Plex, uniFAP, Fantastic Weather, Nanoleaf, LED Simple Effects, Bond Home, Camect.

Posted on
Fri Nov 25, 2016 4:23 pm
Different Computers offline
User avatar
Posts: 2533
Joined: Jan 02, 2016
Location: East Coast

Re: Python x2 and using more CPU?

More info:
Code: Select all
Unknown:~ macadmin$ which python
/usr/bin/python
Unknown:~ macadmin$ python -c "import sys; print sys.executable"
/usr/bin/python
Unknown:~ macadmin$ which -a python
/usr/bin/python
Unknown:~ macadmin$
Got this from looking at things like http://stackoverflow.com/questions/2584 ... v#25858171

SmartThings refugee, so happy to be on Indigo. Monterey on a base M1 Mini w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKitLink, Grafana, Plex, uniFAP, Fantastic Weather, Nanoleaf, LED Simple Effects, Bond Home, Camect.

Posted on
Fri Nov 25, 2016 7:07 pm
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Python x2 and using more CPU?

Looks like some kind of issue with the INDIGOplotD plugin - I'd post over in his forums.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Fri Nov 25, 2016 7:11 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Python x2 and using more CPU?

Different Computers wrote:
Ideas? Is matplotlib really the culprit?

Possibly. Matplotlib doesn't do its own garbage collection, so it depends on how Karl has dealt with this.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Nov 26, 2016 7:14 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Python x2 and using more CPU?

A) yes matplot is a resource hog. 90% of the cpu goes into one function in MATPLOT : plt.savefig() Thats where the actual PNG file gets created
There is a lot of discussion on this topic in MATPLOT forums

B) the memory usage goes up by ~ 100k per plot. Hence the plugin restarts the matplot part every ~ 100 loops to free up the memory at ~ 70Mbyte increased memory usage
This happens despite doing fig.clf() and plt.close(fig). after plt.savefig() to free up memory
I also tried garbage collection with gc.collect() , but that does not seem to help.

So yes it uses a lot of CPU and has memory leaks - that one is covered.


to compare: matplot uses ~ 1-10 seconds cpu (depending on your MAC and features) to create a plot with nice features , GNUPLOT uses less than 0.1 seconds for the same plots


Karl

Posted on
Sat Nov 26, 2016 8:27 am
Different Computers offline
User avatar
Posts: 2533
Joined: Jan 02, 2016
Location: East Coast

Re: Python x2 and using more CPU?

GNUPLOT uses less than 0.1 seconds for the same plots
That seems to suggest GNUPLOT is an alternative, but I haven't seen anything about that here.

SmartThings refugee, so happy to be on Indigo. Monterey on a base M1 Mini w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKitLink, Grafana, Plex, uniFAP, Fantastic Weather, Nanoleaf, LED Simple Effects, Bond Home, Camect.

Posted on
Sat Nov 26, 2016 9:09 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Python x2 and using more CPU?

indigoplotd can be used with MATPLOT and GNUPLOT.. BUT YOU NEED TO INSTALL GNUPLOT. I am using brew to install it.

see http://forums.indigodomo.com/viewtopic.php?f=165&t=15884 in the pdf on how to install it

as for CPU: my old 2008 mac pro ( 2 quad processors) runs easily with MATPLOT never > 30% load with security spy running ... while the 2014 mac mini (16GB i7 2 processors) goes to the limit


Karl

unfortunately next summer the OPSYS will run out of support on the Mac Pro

Posted on
Sat Nov 26, 2016 10:25 am
Different Computers offline
User avatar
Posts: 2533
Joined: Jan 02, 2016
Location: East Coast

Re: Python x2 and using more CPU?

Thanks! I'll give a shot at installing GNUPLOT and reconfiguring IndigoPlotD.

Seems I'm pushing an underpowered Mini with my setup. I'm running Indigo with multiple 3rd party plugins, NO-IP client, and my iTunes AirPlay server on a 2008 2.53 GHz Intel Core 2 Duo that's stock.

SmartThings refugee, so happy to be on Indigo. Monterey on a base M1 Mini w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKitLink, Grafana, Plex, uniFAP, Fantastic Weather, Nanoleaf, LED Simple Effects, Bond Home, Camect.

Posted on
Sat Nov 26, 2016 12:53 pm
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Python x2 and using more CPU?

yes thats a little underpowered..
basic indigo and some simple plugins are ok, but anything with graphics and database activity will stretch it.

Posted on
Tue Jul 04, 2017 8:21 am
hamw offline
Posts: 1212
Joined: Mar 31, 2008

Re: Python x2 and using more CPU?

I am on a 2010 mac mini, and am seeing a lot of CPU usage with indigoplotD, along with the python resource seen in activity monitor. Am using Marplot. is the so solution to change to gnu plot?

Posted on
Tue Jul 04, 2017 8:40 am
kw123 offline
User avatar
Posts: 8333
Joined: May 12, 2013
Location: Dallas, TX

Re: Python x2 and using more CPU?

matplot uses factor of 20 or more CPU than gnu plot. I spend a lot of time on this -- it is a known topic in the matplot discussion forums. The one call that actually does the plot consumes 95% . there is little one can do. Also matplot has a memory leak . but that is handled (restart it self every 100 iterations).

GNUPLOT does 99% of the things the same way. Some special things are different:
- left right line keys (gnu plot has one list, matplot put them left and right)
- line smoothing not as powerful (gnu plot only predefined methods)

Karl

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 2 guests