Server crash

Posted on
Sun Dec 02, 2018 3:18 pm
TwitchCaptain offline
User avatar
Posts: 104
Joined: Dec 13, 2016
Location: San Francisco

Server crash

Indigo crashed for the first time a couple weeks ago. I woke up and my front door was unlocked (because Indigo didn't lock it). It happened again this morning and that prompted me to start doing some digging. I came across this thread, sleuthed down some crash logs and I'm still stumped. There is no crash log for today's crash. The last thing in the indigo log is my Aeotec home energy meter doing a normal update, then the log ends until I restart the app ~8 hours later. The crash a week ago does have a log, but it reads Greek to me.

So, the first question: How do I figure out why Indigo crashed overnight? As in, why isn't there a log, and what other troubleshooting can be performed?

Second question: Do you have a recommend way of restarting Indigo Server automatically? It seems that you have to press buttons in the GUI to fire it up. Is there a CLI method to starting the server?

Last question: Are you interested in the crash log Jay? Indigo 7.2_2018-11-21-153509_hostname.crash - I also have another one that may not be related (I could have been testing stuff at this time?): IndigoPluginHost_2018-11-19-120001_hostname.crash

Thanks!
Last edited by TwitchCaptain on Tue Dec 04, 2018 1:35 pm, edited 1 time in total.

Posted on
Mon Dec 03, 2018 8:36 pm
TwitchCaptain offline
User avatar
Posts: 104
Joined: Dec 13, 2016
Location: San Francisco

Re: First ever server crash?

My crashes seem to be caused by an AppleScript file (a file, not embedded) I run every 4 seconds. Disabling that schedule removes the problem. With the schedule active Indigo is "stopping" (not crashing) once or twice a day. There is nothing in any log to indicate why it's stopping.

Posted on
Tue Dec 04, 2018 11:24 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Server crash

[MODERATOR NOTE] split to a separate topic

Post the AppleScript (inside code tags so it's more readable) and let's see what it's doing.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Dec 04, 2018 1:32 pm
TwitchCaptain offline
User avatar
Posts: 104
Joined: Dec 13, 2016
Location: San Francisco

Re: Server crash

Here's the code. I'll also point out that the log had to be disabled because AppleScript always has troubles finding IndigoServer running. It was enabled during the crashes, but no logs were produced from this script.

Code: Select all
-- Fix Messages.app
-- This will restart the Messages.app if it gets stuck with an error window.
-- Be sure to save as an application and select "stay open after run handler"
-- You may also run this from script editor or from Indigo as a Schedule.

--- Set this to true if you wish for sent messages to be logged in the Indigo Log.
--- If you do not use Indigo, set this to false to avoid errors.
property LogMessages : false
-- When compiled as a standalone app, how often in seconds to check Messages.app.
-- 30 is generally okay, but every 5 seconds is still ok.
property CheckSeconds : 30

on CheckMessages()
   tell application "System Events"
      if (count of (processes whose name is "Messages")) < 1 then
         activate application "Messages"
         my LogIt("Started Messages.app. It was not running.")
         delay 0.5
         my ClearMessages()
      else if (button "Wait" of window 1 of process "Messages" exists) or (button "Ignore Error" of window 1 of process "Messages" exists) then
         my RestartApp("Messages", "It had an error window.")
      else
         try
            -- Check if it's stuck. This is rare, but happens.
            with timeout of 2 seconds
               tell application "Messages" to (count windows)
            end timeout
         on error
            my RestartApp("Messages", "It was stuck.")
         end try
      end if
   end tell
end CheckMessages

on RestartApp(theApp, Msg)
   tell application "System Events"
      set theID to (unix id of processes whose name is theApp)
   end tell
   do shell script "kill -9 " & theID
   delay 0.1
   activate application "Messages"
end RestartApp

on ClearMessages()
   tell application "Messages"
      try
         repeat (count of (get every chat)) times
            delete item 1 of (get every chat)
            tell application "System Events" to tell process "Messages" to keystroke return
         end repeat
      end try
      close every window
   end tell
end ClearMessages

on LogIt(Msg)
   if LogMessages is not true then return
   tell application "IndigoServer" to log Msg
end LogIt

on idle -- Used for standalone app.
   CheckMessages()
   return CheckSeconds
end idle

on run -- Used by Script Editor and Indigo.
   CheckMessages()
end run

Posted on
Wed Dec 05, 2018 10:18 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Server crash

Do you have anything in Indigo that's also trying to talk to Messages? If, for instance, an AppleScript in Indigo is trying to talk to Messages while this script is trying to kill/restart it, I could see how that would cause the server to freeze for some period of time.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Wed Dec 05, 2018 12:44 pm
TwitchCaptain offline
User avatar
Posts: 104
Joined: Dec 13, 2016
Location: San Francisco

Re: Server crash

No, Indigo does not run any other applescripts, just this one, and it's a file not embedded.. I've moved all the Messages.app processing to a Go daemon I wrote (and will release soon). All Indigo does is run a small python script to dump some data off at an API end point in the daemon. That script looks like this, and runs any time the 1023892794 variable changes:
Code: Select all
import urllib
import urllib2
import socket
timeout = 1
socket.setdefaulttimeout(timeout)
subs = urllib.quote(indigo.variables[1891888064].value)
msg = urllib.quote(indigo.variables[1023892794].value)
url = "http://127.0.0.1:8765/api/v1.0/send-message?to="+subs+"&msg="+msg
try:
    urllib2.urlopen(url)
    indigo.server.log(u"Dropped off message with iMessage Relay")
except Exception as err:
    indigo.server.log(u"Error with iMessage-Relay: {}".format(err))


I've adjusted my AppleScript to only run every 30 seconds instead of 4 and no more crashes. I'm probably going to put it back to 4 to see if the crash returns.

Posted on
Wed Dec 05, 2018 1:46 pm
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: Server crash

To be clear, there aren't actually any crashes right? From what I see, the server just hangs up for some amount of time but then continues processing.

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Wed Dec 05, 2018 3:49 pm
matt (support) offline
Site Admin
User avatar
Posts: 21411
Joined: Jan 27, 2003
Location: Texas

Re: Server crash

Can you copy/paste the portion of your Event Log file that shows the last normally logged lines followed by the Indigo Server restarting?

It is strange you don't seem to have any Indigo Server crash report files – just Indigo client and plugin host. If the Indigo Server crashed there should be a crash log.

Lastly, if it does crash after you change the frequency back to every 4 seconds then try commenting out this line in the script:

do shell script "kill -9 " & theID

on the possibility theID is getting set to the Indigo Server process ID. It shouldn't based on the code, but I am suspicious of that kill -9. That would kill the server without any crash report being generated.

Image

Posted on
Wed Dec 12, 2018 8:57 pm
TwitchCaptain offline
User avatar
Posts: 104
Joined: Dec 13, 2016
Location: San Francisco

Re: Server crash

When this issue occurs, I find the Indigo client dialog with two buttons: "start local server" and "connect to remote ...". The server application is just no longer running. All clients (including the iPad on the wall running indigo touch) are disconnected. I have doubly validate this AppleScript as the culprit by running it more often and not running it at all. If it's running there is a chance the app will "die" and it hasn't died yet while the schedule was disabled.

I commented the kill -9 line and replaced it with an echo. This will tell me if the script is somehow getting IndigoServer's pid. Will report back what I find in a few days.
Code: Select all
   do shell script "echo woulda killed " & theID & " > /tmp/check_me.log"
   --   do shell script "kill -9 " & theID


Here's the indigo log leading up to the crash and right after clicking "start local server"
Code: Select all
2018-12-12 15:11:06.936   Z-Wave   received "Mimolite Analog" sensor update to 2800
2018-12-12 15:11:17.878   Trigger   Notify Gate Opened
2018-12-12 15:11:17.880   Trigger   Send Message
2018-12-12 15:11:17.890   Script   Dropped off message with iMessage Relay
2018-12-12 15:11:18.204   Z-Wave   device "Office Light" parameter 23 changed to 1
2018-12-12 15:11:18.504   Z-Wave   device "Master Bathroom Light" parameter 23 changed to 1
2018-12-12 15:11:18.858   Z-Wave   device "Backyard Light" parameter 23 changed to 1
2018-12-12 18:02:01.726   Application   Starting Indigo Server version 7.2.0 (Mac OS X 10.12.6)
2018-12-12 18:02:02.129   Application   Loading attachments
2018-12-12 18:02:02.132   Application   "device target attachment.scpt" script loaded
2018-12-12 18:02:02.133   Application   "group target attachment.scpt" script loaded
2018-12-12 18:02:02.135   Application   "growl attachment.scpt" script loaded
2018-12-12 18:02:02.139   Application   "iTunes attachment.scpt" script loaded
2018-12-12 18:02:02.140   Application   "scenes attachment.scpt" script loaded
2018-12-12 18:02:02.142   Application   "time stamp attachment.scpt" script loaded


The Notify Gate Opened trigger changes a variable when a z-wave command is received. It also changes an LED on some homeseer switches.
The Send Message trigger fires when the variable is changed and runs the python script I posted on this thread previously, embedded.

Hopeful to have more data soon. Thanks!

Posted on
Fri Dec 14, 2018 2:01 am
TwitchCaptain offline
User avatar
Posts: 104
Joined: Dec 13, 2016
Location: San Francisco

Re: Server crash

Code: Select all
server:~ user$ cat /tmp/check_me.log
woulda killed 64770
woulda killed 67711
woulda killed 85320
woulda killed 3601
woulda killed
woulda killed
woulda killed 4088
woulda killed
woulda killed 4239
woulda killed 4286
woulda killed
woulda killed 0
woulda killed 18025


Looks pretty wild in there. kill -9 0 is pretty bad and is likely the culprit. Good catch, Matt!

Page 1 of 1

Who is online

Users browsing this forum: BeeEss and 2 guests