Matplotlib Plugin CSV Engine Issue

Posted on
Sun Jan 13, 2019 5:04 am
mclass offline
Posts: 312
Joined: May 13, 2015
Location: Melbourne, Australia

Matplotlib Plugin CSV Engine Issue

I have been successfully using this plugin on two different installations to plot solar production and consumption of PV systems.

I had a single CSV Engine generating three separate CSV files from device states of the solar system device for the required parameters.

Since upgrading to Matplotlib 0.7.18 one of the installations has been experiencing corruption of the CSV file, a portion of which is shown in the attached image. This in turn generates a plot error as follows:

Code: Select all
   Matplotlib Critical Error       [Solar Prodn iPad Chart] invalid literal for float(): 2085-01-01 00:00:00
   Matplotlib Critical Error       [Solar Prodn iPhone Chart] invalid literal for float(): 2085-01-01 00:00:00


In an effort to track this down, I created three separate CSV engine devices for each parameter, but the problem persists :(

I have not attempted to uninstall the plugin to start afresh, but would like any suggestions as to what may be causing this problem. If re-installation is recommended, some advice as to which files (if any) in addition to the plugin, log, CSV and image files is required
Attachments
Site Consumption CSV File.jpg
Site Consumption CSV File.jpg (346.64 KiB) Viewed 2563 times

Posted on
Sun Jan 13, 2019 6:21 am
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Matplotlib Plugin CSV Engine Issue

mclass wrote:
Since upgrading to Matplotlib 0.7.18 one of the installations has been experiencing corruption of the CSV file, a portion of which is shown in the attached image.

Sorry to hear that you're having trouble. Thanks for reporting the problem.

You're the second user to report this exact issue after upgrading to 0.7.18. I worked extensively with the other user and was unable to find what's causing the error. Our short-term fix was to generate the CSV data for the problem device manually using an external script rather than with the CSV engine. This is obviously not a solution, but a workaround until such time as I'm able to isolate the problem. You might also downgrade to v07.09, although that *may* introduce other problems.

  • the problem was apparently introduced after v0.7.09 (there were no similar reports prior).
  • Both users that have reported the issue have reported it with an energy-related device, although I don't know that's relevant.
  • It appears that the errors are correlated with the introduction of the pandas framework for managing the CSV data.
  • The introduction of pandas is also correlated with the introduction of the duration setting (observation time limit).
  • The errant bit of data looks remarkably like a piece of the Unix Epoch (1970-01-01 00:00:00).
  • I had thought that there might be a Unicode issue because the other user reporting the issue uses Dutch extended Unicode characters in his install, but I'm able to discount that more now.
  • I had thought that this might be a race condition, where two processes were writing to the file simultaneously, but that *shouldn't* be possible (although there may be a bug which makes it possible...)

For folks playing at home, the following command should find any CSV files with a similar errant string in your CSV data:
Code: Select all
 grep -r '/Users/path/to/plugin/csv/logs/' -e '-01-01 00:00:00'

I have a big update coming which I may hold back until I can isolate and fix this issue.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sun Jan 13, 2019 10:43 pm
mclass offline
Posts: 312
Joined: May 13, 2015
Location: Melbourne, Australia

Matplotlib Plugin CSV Engine Issue

Thanks for your prompt response.

I’ll try the rollback later today. Would you suggest that I retain the same plugin preferences file (with my setup) or should I delete everything and start again ie simply install the previous version after deleting the csv and image file?

Thanks again!!


Sent from my iPad using Tapatalk

Posted on
Mon Jan 14, 2019 4:05 am
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Matplotlib Plugin CSV Engine Issue

mclass wrote:
Would you suggest that I retain the same plugin preferences file (with my setup) or should I delete everything and start again ie simply install the previous version after deleting the csv and image file?

  • Disable the plugin within Indigo. This should ensure that everything is written out to the preferences file and all file locks are released.
  • Create a backup of all your CSV data and the plugin preferences file.
  • Extract a copy of 0.7.09 to a separate location.
  • Drag the plugin file onto the Indigo Plugins (Disabled) folder and allow OS X to overwrite the newer version.
  • Re-enable the plugin within Indigo.
I would babysit it for a few cycles to ensure that there are no unexpected gremlins (there shouldn't be, but I can't guarantee that.) If there are a few gremlins, sometimes it's enough just to open the plugin preferences dialog and device dialogs and select save (without changing anything). With respect to the troublesome CSV file, there's no need to delete it--you can manually edit the file with any plain text editor and delete the troublesome text and it should be just fine.

I am testing an updated version of the plugin now and we'll see if that solves the problem. Fingers crossed!

Please let me know how you get on.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Jan 16, 2019 6:06 pm
mclass offline
Posts: 312
Joined: May 13, 2015
Location: Melbourne, Australia

Re: Matplotlib Plugin CSV Engine Issue

Thanks Dave, and apologies for the delay. Hot (southern hemisphere) summer days and a good swell has meant I’ve spent more time surfing the ocean than surfing the web!!

I have totally uninstalled the latest version on one system, and installed V0.7.09 from first principles - it has operated satisfactorily overnight without the CSV issue.

On having more closely scrutinised the issue, it appears that the problematic CSV engine obtained its data from device states produced by Glenn’s Enphase plug-in, whereas the other system (that appears to be behaving itself on the latest Matplotlib plugin version) gets its data from a couple of Ghost XML plugin devices I created for the Fronius system. I am not sure if this is of any assistance to you in tracking down the issue, but .....

I’ll keep a close eye on it and let you know if there’s any other clues I can offer!


Sent from my iPad using Tapatalk

Posted on
Wed Jan 16, 2019 6:55 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Matplotlib Plugin CSV Engine Issue

mclass wrote:
Thanks Dave, and apologies for the delay. Hot (southern hemisphere) summer days and a good swell has meant I’ve spent more time surfing the ocean than surfing the web!!

I have totally uninstalled the latest version on one system, and installed V0.7.09 from first principles - it has operated satisfactorily overnight without the CSV issue.

On having more closely scrutinised the issue, it appears that the problematic CSV engine obtained its data from device states produced by Glenn’s Enphase plug-in, whereas the other system (that appears to be behaving itself on the latest Matplotlib plugin version) gets its data from a couple of Ghost XML plugin devices I created for the Fronius system. I am not sure if this is of any assistance to you in tracking down the issue, but .....

I’ll keep a close eye on it and let you know if there’s any other clues I can offer!


Sent from my iPad using Tapatalk

Thanks for the update. I will check with the other user reporting the same issue to see if his source is from the same plugin.

It was a high of 28ºF here today, with barely a lick of sun. No outdoor surfing going on round these parts..

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Thu Jan 17, 2019 5:51 pm
mclass offline
Posts: 312
Joined: May 13, 2015
Location: Melbourne, Australia

Matplotlib Plugin CSV Engine Issue

Dave,

Rather than run differing versions of Matplotlib (and the possibility of incurring a negative score on my WAF), I have reverted to V0.7.09 on both systems. The CSV engine has performed flawlessly overnight using data from device states from Glenn’s Enphase plugin devices on one system and data from Fronius Ghost XML devices on the other system.

Let me know if you would like me to retry the latest Matplotlib version with either or both systems again. Otherwise, I’m happy to stick with the older revision!

Thanks again for your prompt responses for my pleas for help!!

mclass




Sent from my iPad using Tapatalk

Posted on
Thu Jan 17, 2019 8:26 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Matplotlib Plugin CSV Engine Issue

Thanks for the update. This really is a weird one. I haven't heard back from the other user yet (there's still only two of you), but your latest report makes me believe even more that the issue may lie with the pandas package. Its use makes it much easier to manage from a coding perspective, but it's not really all that helpful if it introduces errors. I looked into preloading an updated version of the library (the one I'm using is kinda old now) but the latest version required newer versions of some other libraries and I felt that domino effect was becoming too risky.

If I can't narrow it down fairly soon, I'll give up on pandas and roll my own.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 2 guests