Plugin Stops Working...Why?

I've taken a stab and making my own plugin. I started with the NOAA plugin and pared it down to do just what I need, which is to request the value of a variable with a Restful call and parse the JSON that is returned. It works perfectly, until it doesn't.

It runs great for several hours, but then stops responding to changing conditions. There is nothing in the log to show where it failed, even though my "try" statements include logging of errors. If I reload the plugin, it has to be forced to quit, according to the log, and then it works again. I can only assume it's just locked up.

What is the best way to debug this problem? Are there any additional debugging tools I can use to determine why it stops working?

Thanks!

[jay (support)]

Hard to tell really what the problem is. Lots of self.debugLog()'s is probably your best bet. If you've got stuff in your runConcurrentThread method, make sure that it doesn't get into a race condition. Also, be careful of how you open/close network connections - that can lead to hangups.

Thank you, Jay!

I will definitely add more logging. I am unsure what a "race condition" is, though. (Edit: Nevermind...Google had the answers to that)

My Restful calls are opening a URL using urllib2, which is what was in the NOAA script. I didn't see where I'd close the connection, though, so that may have something to do with my problem. Maybe I'm stacking up connections like a memory leak.

Thank you, Jay!

I will definitely add more logging. I am unsure what a "race condition" is, though. (Edit: Nevermind...Google had the answers to that)

My Restful calls are opening a URL using urllib2, which is what was in the NOAA script. I didn't see where I'd close the connection, though, so that may have something to do with my problem. Maybe I'm stacking up connections like a memory leak.

Closing:

Code: Select all

import urllib2

f = urllib2.urlopen(url)
response = f.read()
f.close()

However, I have been migrating away from urllib and urllib2 and moving to subprocess calls to Curl (based on advice from this forum.)

Code: Select all

import subprocess

def getTheData(url):
    try:
        proc = subprocess.Popen(["curl", '-vs', url], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        (result, err) = proc.communicate()
        return result
    except:
        indigoServerLog(u"%s" % err)


Dave

[jay (support)]

However, I have been migrating away from urllib and urllib2 and moving to subprocess calls to Curl (based on advice from this forum.)

Really? Why? That seems like a pretty big backwards step to me...

Really? Why? That seems like a pretty big backwards step to me...

From a prior conversation we had:

The subprocess Python library is the one you want to look at. Just make sure that any command you're calling will return in less than about 10 seconds. Embedded Python scripts run in a shared execution environment and any script that takes longer than 10 seconds is killed so as to not slow other executions. If it takes longer, just execute it as a separate file so it'll get its own process and can run as long as you want.

In all seriousness though, I was under the impression that using the subprocess library was more stable than the urllib/2 libraries overall. Is that not the case? I want to do as much as I can to be smart, stable and secure as I can be given the task at hand.

Dave

Thank you very much, Dave. I suspect that is exactly what has been my problem. I was opening the URL and reading from it, but never closing it. I've made the change and will see if it's still running by morning.

[jay (support)]

The subprocess Python library is the one you want to look at. Just make sure that any command you're calling will return in less than about 10 seconds. Embedded Python scripts run in a shared execution environment and any script that takes longer than 10 seconds is killed so as to not slow other executions. If it takes longer, just execute it as a separate file so it'll get its own process and can run as long as you want.

That's when you're running from a script - there is no timeout from inside a plugin...

In all seriousness though, I was under the impression that using the subprocess library was more stable than the urllib/2 libraries overall. Is that not the case? I want to do as much as I can to be smart, stable and secure as I can be given the task at hand.

I've never heard that either urllib was unreliable - we use it all over the place and haven't had any issues. Not using them increases code size and complexity, which (IMO) is more likely to cause issues. But that's just me - I'm pretty lazy, so if something takes less coding on my part I'm all for it.

[jay (support)]

Code: Select all

import urllib2

f = urllib2.urlopen(url)
response = f.read()
f.close()

One of the implementation details of this particular method and the object it returns is that when the object goes out of scope (is deleted) the connection is automatically closed. In the NOAA plugin, we always use urlopen in a way that reduces the scope of the returned object: in the update() method (which gets called frequently and is only briefly in scope) and in validateDeviceConfigUi() which, again, is only in scope briefly. For that reason, we don't specifically close the connection (since at the end of the method run it will automatically close).

However, if you're calling it repeatedly in something that stays in scope, such as in the runConcurrentThread() method then you're going to leak connections and you should close it (or explicitly delete it, which will close it).

I'm going to rev the NOAA plugin so that it explicitly closes the connection so that if anyone uses it as a starting point it will be clearer so they can avoid this potential issue.

However, I have been migrating away from urllib and urllib2 and moving to subprocess calls to Curl (based on advice from this forum.)

You should look into using the Requests library IMO... it has a better API than urllib and urllib2 and can be included in your plugin (without requiring the user to install anything externally.) Of course, this works best with API .19 so you get the newer Python.

Going out to a subprocess and curl seems like it would be WAY less efficient, by a big factor. Curl still has to do the actual network communication but now you are requiring the system to spin up a brand new execution context.

I made the change last night to explicitly close my connection and I thought it fixed the problem. It did not. After about six hours, the plugin locked up again. It polls the URL every five seconds, so it looks like it ran successfully for just over 4,000 calls. Maybe 4096, which could be a useful bit of information? Anyway, I kicked it off again this morning with a counter that tracks the number of successful calls and writes it to the log. In the meantime, it seems I'm still no closer to figuring out why it locks up on me. When I get back to that machine, I'll post up the code and, hopefully, one of you more-knowledgeable folks will see where my problem lies.

[jay (support)]

Just to clairfy what @RogueProeliator said, the requests lib requires Python 2.6, so basically it's likely to not work on any Indigo version earlier than 6.1.0. For simple things like just getting the contents of a URL, urllib2 is extremely easy to use and is how I'd do it.

The requests library is, however, a great library and we're looking into including it in Indigo 7.

Well, the plugin only successfully updated 2557 times this last run.

This plugin is meant to support a little project I took on over the holidays where I've set up an ESP8266 to run as a Restful server that controls a relay. That relay is connected to my garage door opener so I can open and close it remotely via Indigo. There is a microswitch installed on the track that is closed when the garage door is closed, so this plugin polls the device and updates the state such that TRUE means the garage door is closed. It should be dead-simple to poll the URL and parse the JSON result repeatedly, but I just can't see why it keeps stopping. Once I get this all running properly, I'll document it and start a thread about it, in case others are interested in doing something similar.

Here is the code as it currently stands. Can anyone see why it's locking up?

Code: Select all

#! /usr/bin/env python
# -*- coding: utf-8 -*-
####################
# Copyright (c) 2014, Perceptive Automation, LLC. All rights reserved.
# http://www.indigodomo.com
#

################################################################################
# Imports
################################################################################
import indigo
import urllib2
import json
from xml.dom.minidom import parseString

################################################################################
# Globals
################################################################################
theUrlBase = u"http://"


########################################
def updateVar(name, value, folder=0):
   if name not in indigo.variables:
      indigo.variable.create(name, value=value, folder=folder)
   else:
      indigo.variable.updateValue(name, value)

################################################################################
class Plugin(indigo.PluginBase):
   ########################################
   # Class properties
   ########################################
   
   ########################################
   def __init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs):
      indigo.PluginBase.__init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs)
      self.debug = pluginPrefs.get("showDebugInfo", False)
      self.deviceList = []
   
   ########################################
   def __del__(self):
      indigo.PluginBase.__del__(self)
   
   ########################################
#    def goToStationURL(self, valuesDict, typeId, devId):
#       self.browserOpen("http://" + device.pluginProps["address"] + "/digital/0")
      
   ########################################
   def deviceStartComm(self, device):
      self.debugLog("Starting device: " + device.name)
      if device.id not in self.deviceList:
         self.update(device)
         self.deviceList.append(device.id)
         device.updateStateImageOnServer(indigo.kStateImageSel.Auto)

   ########################################
   def deviceStopComm(self, device):
      self.debugLog("Stopping device: " + device.name)
      if device.id in self.deviceList:
         self.deviceList.remove(device.id)
         
   ########################################
   def runConcurrentThread(self):
      self.debugLog("Starting concurrent thread")
      successfulUpdates = 0
      try:
         while True:
            self.sleep(5)
            # now we cycle through each device
            for deviceId in self.deviceList:
               # call the update method with the device instance
               self.update(indigo.devices[deviceId])
            successfulUpdates = successfulUpdates + 1
            self.debugLog("Successful Updates: " + str(successfulUpdates))
      except self.StopThread:
         pass
   
   ########################################
   def update(self,device):
      #self.debugLog("Updating device: " + device.name)
      # download the file
      theUrl = theUrlBase + device.pluginProps["address"] + "/digital/0"
      try:
         thisFile = urllib2.urlopen(theUrl)
         readData = json.load(thisFile)
         thisFile.close()
         doorStatus = readData['return_value']
         
         #self.debugLog("doorStatus: " + str(doorStatus))
         
         if (doorStatus != device.states["doorStatus"]):
            device.updateStateOnServer(key="doorStatus", value=doorStatus, clearErrorState=True)

      except urllib2.HTTPError, e:
         self.errorLog("HTTP error getting station %s data: %s" % (device.pluginProps["address"], str(e)))
         return
      except Exception, e:
         self.errorLog("Unknown error getting station %s data: %s" % (device.pluginProps["address"], str(e)))
         return
         
      #   self.updateDeviceState(device, state, doorStatus)

   ########################################
   def updateDeviceState(self,device,state,newValue):
      if (newValue != device.states[state]):
         # if state == "doorStatus":
#             newValue = newValue.split(".")[0]
         device.updateStateOnServer(key=state, value=newValue, clearErrorState=True)

   ########################################
   # UI Validate, Close, and Actions defined in Actions.xml:
   ########################################
   def validateDeviceConfigUi(self, valuesDict, typeId, devId):
      #theUrl = theUrlBase + (device.pluginProps["address"]) + "/digital/0"
      
      stationId = valuesDict['address'].encode('ascii','ignore').upper()
      valuesDict['address'] = stationId
      theUrl = theUrlBase + stationId + "/digital/0"   
      
      try:
         urllib2.urlopen(theUrl)
      except urllib2.HTTPError, e:
         errorsDict = indigo.Dict()
         errorsDict['address'] = "Station not found or isn't responding"
         self.errorLog("Error getting station data: %s" % (str(e)))
         return (False, valuesDict, errorsDict)
      return (True, valuesDict)
      
   ########################################
   # Menu Methods
   ########################################
   def toggleDebugging(self):
      if self.debug:
         indigo.server.log("Turning off debug logging")
         self.pluginPrefs["showDebugInfo"] = False
      else:
         indigo.server.log("Turning on debug logging")
         self.pluginPrefs["showDebugInfo"] = True
      self.debug = not self.debug



Thanks

You should look into using the Requests library IMO... it has a better API than urllib and urllib2 and can be included in your plugin (without requiring the user to install anything externally.)

The requests library is, however, a great library and we're looking into including it in Indigo 7.

Is including the Requests library within your plugin (ie copying it into the plugin folder and importing from there) approved and/or acceptable then? That's what I've done in Evohome, but it was only meant to be temporary during alpha builds!


Sent from my iPhone using Tapatalk

I've never heard that either urllib was unreliable - we use it all over the place and haven't had any issues. Not using them increases code size and complexity, which (IMO) is more likely to cause issues. But that's just me - I'm pretty lazy, so if something takes less coding on my part I'm all for it.

Where this all started for me was some rare cases where urllib2 would return an error that would throw an "invalid literal for int() with base 16". A user very helpfully suggested that curl would skate around this problem (which to date has worked to solve that particular error completely.)

But the idea that using curl would consume more resources makes sense too. If I can use urllib2 and make things peppier, that's what I'll do. I'll probably still use curl in the WUnderground plugin as that's where the rare error has shown up...so far.

Dave