Starting and communicating with a background process

Posted on
Mon Jan 11, 2021 2:56 pm
FlyingDiver offline
User avatar
Posts: 5335
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Starting and communicating with a background process

From a plugin.

Anyone got some example code? I looked at some likely suspects, but didn't find anything useful. HBK just starts/stops home bridge instances, I couldn't find any communication while running. I found a couple others that run a process once and do something with the return code or output, but I need a daemon type process that I can get output from indefinitely.

This is what happens when the ONLY working API for a device is Python3 code. :(

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Mon Jan 11, 2021 7:59 pm
aaronlionsheep offline
Posts: 129
Joined: Feb 24, 2019
Location: Virginia, USA

Re: Starting and communicating with a background process

I don't have any of my own examples I can provide, but are you familiar with the subprocess module in Python? It sounds like that may be what you want. You can spawn a new process from Python 2.7 code (to run your Python 3 script) and set it up such that the stdin and stdout can be read and written to.

This is the Python 2.7 documentation for the subprocess module: https://docs.python.org/2.7/library/subprocess.html

And I think this site has a great writeup with some examples on using the module. It covers writing to and reading from stdin and stdout of a subprocess: https://lyceum-allotments.github.io/2017/03/python-and-pipes-part-5-subprocesses-and-pipes/

Here is a snippet taken from one of the examples where you can write to the stdin of the spawned process (you can do something similar for reading from stdout as well):

Code: Select all
proc = subprocess.Popen(["python", "say_my_name.py"], stdin=subprocess.PIPE)

proc.stdin.write("matthew\n")
proc.stdin.write("mark\n")
proc.stdin.write("luke\n")


Maybe this will point you in the right direction or at least make you aware of the subprocess module if you haven't heard of it. Good luck!

Posted on
Mon Jan 11, 2021 8:00 pm
FlyingDiver offline
User avatar
Posts: 5335
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Starting and communicating with a background process

I've actually gotten the code written to start up the other process, and it's writing to stdout, but reading from the pipe doesn't seem to be working. :(

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Mon Jan 11, 2021 8:10 pm
aaronlionsheep offline
Posts: 129
Joined: Feb 24, 2019
Location: Virginia, USA

Re: Starting and communicating with a background process

Are you using a for loop with "proc.stdout" to iterate over each line of stdout? If so, don't forget that the stdout variable isn't fully populated until the process finishes. I believe you have to iterate over "proc.stdout.readline" to read from stdout while the process is running. Something like:

Code: Select all
for out_line in iter(proc.stdout.readline):
    print(out_line)


Or I think calling the readline() function on stdout will simply block until it can supply an entire line of stdout data.

Posted on
Mon Jan 11, 2021 8:11 pm
FlyingDiver offline
User avatar
Posts: 5335
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Starting and communicating with a background process

Code: Select all
        def reader():
            self.logger.debug("{}: Reader thread starting...".format(device.name))
            p = Popen(["/usr/bin/python3", "camect/__init__.py", "{}:{}".format(self.address, self.port), self.username, self.password], stdout=PIPE, bufsize=1)
            self.logger.debug("{}: Reader process started: {}".format(device.name, p))
            with p.stdout:
                for line in iter(p.stdout.readline, b''):
                    self.logger.threaddebug("websocket received event: {}".format(line))
                    indigo.activePlugin.processReceivedEvent(self.deviceID, line)

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Posted on
Mon Jan 11, 2021 8:26 pm
aaronlionsheep offline
Posts: 129
Joined: Feb 24, 2019
Location: Virginia, USA

Re: Starting and communicating with a background process

So I got it working. It appears that you need to tell the spawned python process to not buffer its output by passing the "-u" flag. My demo below works with the "-u" and it does nothing at all without the flag.

Here is my quick example:

Code: Select all
[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% cat runner.py
import time


def main():
    while True:
        print(time.time())
        time.sleep(1)


if __name__ == "__main__":
    main()


Code: Select all
[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% cat executor.py
from subprocess import Popen, PIPE


def main():
    print("Starting runner...")
    p = Popen(["/usr/bin/python3", "-u", "runner.py"], stdout=PIPE, bufsize=1)  # Note the "-u" flag...
    print("Runner started! Runner output follows...")

    with p.stdout:
        for line in iter(p.stdout.readline, b''):
            print(line.strip())


if __name__ == "__main__":
    main()


And the current time from runner.py will be displayed in the executor.py script:
Code: Select all
[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% python executor.py
Starting runner...
Runner started! Runner output follows...
1610418164.4193301
1610418165.423
1610418166.427841
1610418167.431348
1610418168.433428
^CTraceback (most recent call last):
  File "runner.py", line 11, in <module>
Traceback (most recent call last):
  File "executor.py", line 15, in <module>
    main()
  File "executor.py", line 10, in main
    for line in iter(p.stdout.readline, b''):
KeyboardInterrupt
    main()
  File "runner.py", line 7, in main
    time.sleep(1)
KeyboardInterrupt


I have a python script invoked with Python 2.7.15 spawning another python script using Python 3.8.5, so it matches your scenario.

Code: Select all
[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% python --version
Python 2.7.15
[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% python3 --version
Python 3.8.5

Posted on
Mon Jan 11, 2021 8:42 pm
FlyingDiver offline
User avatar
Posts: 5335
Joined: Jun 07, 2014
Location: Southwest Florida, USA

Re: Starting and communicating with a background process

I'll keep that in mind. Right now I'm testing some changes the Camect team sent me using the direct websocket connection. If I can get that to work I won't need the subprocess method.

joe (aka FlyingDiver)
my plugins: http://forums.indigodomo.com/viewforum.php?f=177

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 1 guest