Need help with plots

Posted on
Tue Apr 21, 2020 7:08 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Need help with plots

I have created three variables: hourlyGallonscurrent, gallonscurrent, and priorhourlygallons

At the beginning of each hour (starting at 1 AM) I get the counter value from my water counter (I have checked this and its working properly) and then calculate the difference in the counter value from the prior hour. Then the difference is placed into hourly gallons used and then that value is used for the next hours "priorhour" calculation. I then update the gallons current for that hour to be the total for the day up to that point. At midnight I zero out the values and start again the next day

I am attempting to create two charts:

1) shows an hourlyusage throughout the day,

For the hours chart I used hourlygallonscurrent as the variable refreshed every hour with 24 bars, scale every day. What I get is figure in light blue below (First figure). What I want is figure similar to dark green hourly chart below (Figure 2)?

2) a monthly chart showing the total used each day. I am plotting gallonscurrent
For the daily chart I am plotting gallonsused refreshed everyday with 31 bars and scale every month. What I get is the bars in lt blue (larger plot) - Figure 3.

I want the plot below that shows things in days as one bar total for the day. If the plot is generated in the middle of the day then I want the gallons used so far for the day to be shown like the monthly chart below - Figure4.

I must be doing something basically wrong but not sure what?
Attachments
Screen Shot 2020-04-21 at 8.52.04 PM.png
Screen Shot 2020-04-21 at 8.52.04 PM.png (42.9 KiB) Viewed 4072 times
Screen Shot 2020-04-21 at 8.41.35 PM.png
Screen Shot 2020-04-21 at 8.41.35 PM.png (36.14 KiB) Viewed 4072 times
Screen Shot 2020-04-21 at 8.57.58 PM.png
Screen Shot 2020-04-21 at 8.57.58 PM.png (77.1 KiB) Viewed 4072 times
Screen Shot 2020-04-21 at 8.41.42 PM.png
Screen Shot 2020-04-21 at 8.41.42 PM.png (39.64 KiB) Viewed 4072 times

Posted on
Tue Apr 21, 2020 8:16 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

Happy to try to help. Bar charts are a little different than other chart types due to the fact that they don't do a great job of displaying a ton of observations. That is not to say that you're trying to plot too many--rather, it explains why bar chart settings are handled so differently. For the sake of argument, I'll assume that your data files are properly configured:

Code: Select all
Timestamp,column name
2020-04-18 21:07:55.449262,66.4
2020-04-18 21:23:09.511970,66.1
2020-04-18 21:38:23.964526,66.4
2020-04-18 21:53:38.086074,66.1
2020-04-18 22:08:52.838145,65.8
2020-04-18 22:24:06.786853,65.8
2020-04-18 22:39:20.730052,65.7
2020-04-18 22:54:34.931717,65.7

Taking your first example, if your CSV data looks like this:
Code: Select all
Timestamp,column name
2020-04-18 07:07:55.449262,42
2020-04-18 21:23:09.511970,96

Then I believe what Matplotlib is doing is interpolating the intermediate bars. I think what you'll want is a data file that looks like this:
Code: Select all
Timestamp,column name
2020-04-18 07:07:55.449262,42
2020-04-18 08:07:55.449262,0
2020-04-18 09:07:55.449262,0
...
2020-04-18 19:07:55.449262,0
2020-04-18 20:07:55.449262,0
2020-04-18 21:23:09.511970,96

I think the same might be the case for your second example. If I've somehow missed the mark, please share some examples of your data and it'll help me better visualize what's going on.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Apr 22, 2020 3:14 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

Dave:
Thanks for your help. Here is the csv snapshot. The number are not going to zero for the hours when no water usage used. If there is a better way to show water usage than bar graphs please enlighten me so I can use it. I just need to know which hours it is being used and how much each hour and each day. I have included a snapshot of my configuration for daily and hourly. Interestingly enough I can get Grafana to work somewhat, just that the data is shifted incorrectly. In other word the bars are all 1 hour too late and the day graph is one day too late I would have thought that for the 21 of April the water used would post to that day but instead it posts to the 22? (I have included that graph also. The blue and pourple chart is from grafana not MAtplot...

Thanks again fro your help.
Attachments
Screen Shot 2020-04-22 at 5.09.19 PM.png
Screen Shot 2020-04-22 at 5.09.19 PM.png (116.77 KiB) Viewed 4025 times
Screen Shot 2020-04-22 at 5.10.24 PM.png
Screen Shot 2020-04-22 at 5.10.24 PM.png (120.55 KiB) Viewed 4025 times
Screen Shot 2020-04-22 at 5.12.57 PM.png
Screen Shot 2020-04-22 at 5.12.57 PM.png (50.82 KiB) Viewed 4025 times
Screen Shot 2020-04-22 at 5.05.32 PM.png
Screen Shot 2020-04-22 at 5.05.32 PM.png (191.35 KiB) Viewed 4025 times
Screen Shot 2020-04-22 at 5.06.05 PM.png
Screen Shot 2020-04-22 at 5.06.05 PM.png (149.4 KiB) Viewed 4025 times

Posted on
Wed Apr 22, 2020 5:32 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

It's a little confusing for sure, but the settings in the plugin work a little differently than you might expect.

The X and Y settings are meant to format and scale the axes, but they don't directly impact the data that's displayed. When you set the number of bars to 24, the plot is going to take the last 24 observations from the data set. If you want to plot 24 hours with one bar per hour, then you'll need to save your data every hour. What you're getting is the last 24 observations (6 hours).

Does that makes sense?

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Apr 22, 2020 7:44 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

So if I take readings every hour and want 24 hours worth of data on 1 bar

So No of Bars 1, Under X axis Scale every hour, days to show 31, but this only give me one day?

Getting closer. This is correct for just the day
Now how do I get all 31 days of the month on one chart?

This chart gave me just one day for the month.

Ok I reset the data input so now I will wait and recheck tomorrow and see the result. thanks for your help. and we will see if Ihave it correct tomorrow
Attachments
Screen Shot 2020-04-22 at 9.40.30 PM.png
Screen Shot 2020-04-22 at 9.40.30 PM.png (37.36 KiB) Viewed 4004 times
Screen Shot 2020-04-22 at 9.40.50 PM.png
Screen Shot 2020-04-22 at 9.40.50 PM.png (113.61 KiB) Viewed 4004 times

Posted on
Wed Apr 22, 2020 8:17 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

The plugin doesn't do any combining of observations--like distilling hourly observations into a daily value--so you'll have to do that when saving the data to CSV. Setting the number of bars to 1 will just plot the latest single observation in the CSV file.

Say your data file contains the following:
Code: Select all
timestamp,value
2020-04-01 00:00:00.000000,123
2020-04-02 00:00:00.000000,456
2020-04-03 00:00:00.000000,789
...
2020-04-28 00:00:00.000000,123
2020-04-29 00:00:00.000000,456
2020-04-30 00:00:00.000000,789

If you set the number of bars to 1, you'll just get the data for 4/30.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri Apr 24, 2020 2:44 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

ok Nearly perfect. Here is the hourly plot working almost like I need it except for two issues. The data is from MN to 1609 pm, but it only start charting the data from the first value grater than 0? Also the data is off by one hour, for example at 11:20 there was 5.5 gallons used, at 12:08 pm there was 17.5 gallon, 1:34 19 gallons used and at 3:56 pm there was 54 gallons used. The plot is plotting them for the following hour instead of the hour they occurred in. Also how do I get the hours of 0 plotted. See csv and plot, settings.
Attachments
Screen Shot 2020-04-24 at 4.42.29 PM.png
Screen Shot 2020-04-24 at 4.42.29 PM.png (149.53 KiB) Viewed 3963 times
Screen Shot 2020-04-24 at 4.42.16 PM.png
Screen Shot 2020-04-24 at 4.42.16 PM.png (66.28 KiB) Viewed 3963 times
Screen Shot 2020-04-24 at 4.37.25 PM.png
Screen Shot 2020-04-24 at 4.37.25 PM.png (92.21 KiB) Viewed 3963 times
Screen Shot 2020-04-24 at 4.31.47 PM.png
Screen Shot 2020-04-24 at 4.31.47 PM.png (66.81 KiB) Viewed 3963 times

Posted on
Fri Apr 24, 2020 3:58 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

With respect to the plugin not plotting the leading zero values, I'll need to look into that some more. This is not something that I actively sought to do, and it may be a default behavior of Matplotlib. There may be a way for me to override this behavior. Let me see what I can do.

Regarding the plot being off by an hour, what I'm seeing is that the chart is showing what the data are showing. The data is showing 5.5 gallons at 12:09 and the chart is showing 5.5 at 12:09 (the bar is centered on the plot time). Are you generating the CSV data with the plugin?

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri Apr 24, 2020 4:14 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

Maybe if I let it go another day it will post the zero values as it did for 3 oo? I will wait and see. Any suggestions on how to plot the data another way other than bar graph to post the consumption for the prior hour?

Posted on
Fri Apr 24, 2020 4:33 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

Dave one other thought, can I force the plugin to check at a certain time. Ie if I check at 1:59,,2:59 .... the. It would
Likely post in the correct hour. Right now it’s checking at 08 which is probably the time I start the plugin

Posted on
Fri Apr 24, 2020 5:04 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

There is no setting to key the refresh time in the CSV engine to a certain time. It does indeed proceed from the last time it was restarted.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Fri Apr 24, 2020 5:08 pm
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

Maybe I can make a schedule to start every night at 11:59 pm. That would work for both the day and hours plots

Posted on
Fri Apr 24, 2020 5:17 pm
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

That would work, but it would drift over time. Restarting every day would probably keep the drift to a minimum.

Getting all CSV data to refresh on a precise schedule within the plugin is non-trivial because the devices "get in line" for their refreshes.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Apr 25, 2020 5:42 am
DaveL17 offline
User avatar
Posts: 6741
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Need help with plots

After doing some research on this, it does indeed appear that Matplotlib skips plotting leading zero values when drawing bar charts. There's some debate as to whether it's a bug or a feature, but nonetheless, it is reality. The chart below was constructed by setting the first value in the series to something very small (in this case 0.1) such that the bar is plotted but effectively not visible. Note that I also changed the scale of the X axis bins to every four hours to make the axis labels more readable.

I will look into whether an optional setting could be created to adjust the data slightly such that the user can elect to have leading zero values plotted (or not), or somehow force the chart to display the desired timeline regardless of the values of the data.. Stay tuned.

chart_farberm_bar.png
chart_farberm_bar.png (12.11 KiB) Viewed 3890 times

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Apr 25, 2020 6:47 am
farberm offline
Posts: 393
Joined: Feb 24, 2008

Re: Need help with plots

Thanks... I could always set the default value to 0.01 and then add, that way it is never zero. One other option I was thinking about was doing some scripting...

What I ideally want to do is display a calendar and the user selects a day from the calendar. The script gets the 24 values for that day (plus the first entry for the next day) from the csv file and creates a new file used to plat data ( I actually would get 25 data points.) I would then manipulate the data to delete the first data point and then plot the next 24 data points as each nours consumption. That would cause a shift in data to one hour earlier. I think it would work that way but I will need to elicit my niece for help to do the python programming cuz it will be far beyond my capabilities. I am trying to replace my failing irrigation caddy watering system -(see below) I do not know if there is a way to look at the operating programming on the irrigation caddy. I tried to contact the developer - no response) It has a web interface but not sure how to download the programming behind it. No API published that I can find.
Attachments
Screen Shot 2020-04-25 at 8.42.08 AM.png
Screen Shot 2020-04-25 at 8.42.08 AM.png (101.45 KiB) Viewed 3886 times

Who is online

Users browsing this forum: No registered users and 1 guest

cron