Variable Value Duplication Issue

Posted on
Mon Aug 19, 2019 1:05 pm
InsteonDiego offline
Posts: 29
Joined: Dec 07, 2016

Variable Value Duplication Issue

Hi Jay -

Not sure exactly where to post this so hopefully I'm in the right place. I am seeing an issue with 7.3.1 with variable value duplication. Scenario: I use the Better Email plugin to scrape data from a daily email and then do a regex match to place this data into a variable for further processing. Every single time I reboot the Mac Indigo Server, I see this value duplicated in the postgres database with the same value and updated timestamp of the Indigo Server start. I figured it was an issue with the plugin, so I disabled the trigger and restarted. Still received a duplicate. I then disabled the plugin all together and restarted and still a duplicate. I'm not sure what else could be causing the issue as the only entrance for data addition to this variable is via the Better Email plugin. Any ideas?

Example (I rebooted twice this morning for testing):
26 "2019-08-14 08:06:14.138454" "1.79"
30 "2019-08-15 08:36:09.326763" "1.27"
31 "2019-08-16 08:06:08.970459" "0.96"
32 "2019-08-17 09:21:08.484283" "1.24"
33 "2019-08-18 09:06:09.260185" "1.25"
36 "2019-08-19 08:06:13.431742" "1.62"
37 "2019-08-19 11:48:49.429926" "1.62"
38 "2019-08-19 11:54:57.634342" "1.62"

Posted on
Tue Aug 20, 2019 9:15 am
jay (support) offline
Site Admin
User avatar
Posts: 15858
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Variable Value Duplication Issue

Just to clarify: the lines you pasted above are entries from the SQL Logger plugin to a Postgres database? And every time you restart the server you see a variable update entry?

If so then a couple of other questions:

  1. Is that variable the only one that's getting a duplicate entry when the server is restarted?
  2. Are you positive that something else isn't updating that variable?

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Aug 20, 2019 2:05 pm
InsteonDiego offline
Posts: 29
Joined: Dec 07, 2016

Re: Variable Value Duplication Issue

Thanks Jay for your response. The lines I pasted above are from PGAdmin a gui view into the Postgres database. And yes, EVERY single time I restart the server I see the duplicate. I only became aware of it because the associated IndigoPlotD would crash since it only expected a single value per day. It took me a few times to realize what was going on.

1. I've quite a few variables so not simple to tell but I don't see any clear duplicates. I am happy to run any queries or check anything specific that would be helpful in this regard.
2. I have no idea what else would update the variable nor why it would occur EVERY time I restart.

Posted on
Tue Aug 20, 2019 3:32 pm
jay (support) offline
Site Admin
User avatar
Posts: 15858
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Variable Value Duplication Issue

If it's not happening to every variable, but only that one, then it seems pretty clear that something is causing the variable to update (outside of Indigo itself). Can you correlate exactly what time the variable update date/time is vs what the actual server restart time is (from the Indigo log file)? I'm wondering when exactly the update is getting made compared to the startup (or perhaps the shutdown) time of the Indigo Server.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Aug 20, 2019 3:58 pm
InsteonDiego offline
Posts: 29
Joined: Dec 07, 2016

Re: Variable Value Duplication Issue

With some further testing, I realized I can now duplicate the value simply by stopping and restarting the Indigo server, no reboot required. Here is the output of the startup:

Aug 20, 2019 at 2:51:08 PM
Trigger Indigo Server Started
Trigger Interface Connection Initialized
Connected to PowerLinc 2413U interface on port usbserial-A603390P
PowerLinc address 2F.F8.E5, firmware version 9.11
Trigger Interface Connection Initialized
SQL Logger updating all variable tables
SQL Logger update complete
Alexa-Hue Bridge Alexa-Hue Bridge checking network access by attempting to access 'www.google.com'
Alexa-Hue Bridge Alexa-Hue Bridge network access check to www.google.com successfully completed.
Email Sent to "xxx@xxx.com"; subject "Indigo Server Started"
Email Sent to "xxx@xxx.com"; subject "Interface Connection Initialized"
Email Sent to "xxx@xxx.com"; subject "Interface Connection Initialized"
INDIGOplotD checking py-restore files
utilities versionCheck for com.karlwachs.utilities installed: 1.18.26
utilities the version of the plugin is up to date; version on server:1.18.25

And note the entry in the database two seconds later:

39 "2019-08-20 09:21:10.863594" "1.22"
40 "2019-08-20 14:51:10.234324" "1.22"

Posted on
Tue Aug 20, 2019 5:12 pm
matt (support) offline
Site Admin
User avatar
Posts: 19438
Joined: Jan 27, 2003
Location: Texas

Re: Variable Value Duplication Issue

I think what you are seeing is that anytime the SQL Logger plugin starts it writes to the database all the current variable values (and device states if I remember correctly). It does this because the plugin doesn't know if the variable value (or device state) has changed since it was last started, so it always pushes the current values to the database. It doesn't try to query out of the database the last value to see if it really changed, unfortunately. The logic here though is that it is better to have a duplicate than to have the wrong value.

Image

Posted on
Tue Aug 20, 2019 5:22 pm
InsteonDiego offline
Posts: 29
Joined: Dec 07, 2016

Re: Variable Value Duplication Issue

Thank you Matt, this makes complete sense. Basically, this has been happening forever, but I never noticed it as I wasn't graphing data that expected a single value per day. I'll work with Karl to see if we can find a way for IndigoPlotD to deal with this scenario,

Posted on
Tue Aug 20, 2019 5:29 pm
InsteonDiego offline
Posts: 29
Joined: Dec 07, 2016

Re: Variable Value Duplication Issue

Matt - As I think about this more, this could be problematic when working with daily data. Suppose I get a value at 10am today, but restart the server tomorrow at 8am, before the new data comes in. If I am then forced to select a single value for the day, how would I programmatically know which is the old value and which is new (I guess I could try comparison from the day before, but that are several cases where that would be error-prone). Are there any other thoughts on how to "remove" this additional value (via trigger) or better yet, prevent it from happening for certain variables? Perhaps some type of exclusion list for SQL logger?

Posted on
Tue Aug 20, 2019 6:12 pm
FlyingDiver online
Posts: 3587
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Variable Value Duplication Issue

The client (plugin) that's querying the database should be responsible for making sure it doesn't crash and gets the appropriate data. If only one value is wanted, it needs to do that correctly.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Tue Aug 20, 2019 7:13 pm
jay (support) offline
Site Admin
User avatar
Posts: 15858
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Variable Value Duplication Issue

FlyingDiver wrote:
The client (plugin) that's querying the database should be responsible for making sure it doesn't crash and gets the appropriate data. If only one value is wanted, it needs to do that correctly.


Exactly - it's the responsibility of the data consumer to deal appropriately with the scenario.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 1 guest