Starting and communicating with a background process

From a plugin.

Anyone got some example code? I looked at some likely suspects, but didn't find anything useful. HBK just starts/stops home bridge instances, I couldn't find any communication while running. I found a couple others that run a process once and do something with the return code or output, but I need a daemon type process that I can get output from indefinitely.

This is what happens when the ONLY working API for a device is Python3 code.

I don't have any of my own examples I can provide, but are you familiar with the subprocess module in Python? It sounds like that may be what you want. You can spawn a new process from Python 2.7 code (to run your Python 3 script) and set it up such that the stdin and stdout can be read and written to.

This is the Python 2.7 documentation for the subprocess module: https://docs.python.org/2.7/library/subprocess.html

And I think this site has a great writeup with some examples on using the module. It covers writing to and reading from stdin and stdout of a subprocess: https://lyceum-allotments.github.io/2017/03/python-and-pipes-part-5-subprocesses-and-pipes/

Here is a snippet taken from one of the examples where you can write to the stdin of the spawned process (you can do something similar for reading from stdout as well):

Code: Select all

proc = subprocess.Popen(["python", "say_my_name.py"], stdin=subprocess.PIPE)

proc.stdin.write("matthew\n")
proc.stdin.write("mark\n")
proc.stdin.write("luke\n")


Maybe this will point you in the right direction or at least make you aware of the subprocess module if you haven't heard of it. Good luck!

I've actually gotten the code written to start up the other process, and it's writing to stdout, but reading from the pipe doesn't seem to be working.

Are you using a for loop with "proc.stdout" to iterate over each line of stdout? If so, don't forget that the stdout variable isn't fully populated until the process finishes. I believe you have to iterate over "proc.stdout.readline" to read from stdout while the process is running. Something like:

Code: Select all

for out_line in iter(proc.stdout.readline):
    print(out_line)


Or I think calling the readline() function on stdout will simply block until it can supply an entire line of stdout data.

Code: Select all

        def reader():
            self.logger.debug("{}: Reader thread starting...".format(device.name))
            p = Popen(["/usr/bin/python3", "camect/__init__.py", "{}:{}".format(self.address, self.port), self.username, self.password], stdout=PIPE, bufsize=1)
            self.logger.debug("{}: Reader process started: {}".format(device.name, p))
            with p.stdout:
                for line in iter(p.stdout.readline, b''):
                    self.logger.threaddebug("websocket received event: {}".format(line))
                    indigo.activePlugin.processReceivedEvent(self.deviceID, line)


So I got it working. It appears that you need to tell the spawned python process to not buffer its output by passing the "-u" flag. My demo below works with the "-u" and it does nothing at all without the flag.

Here is my quick example:

Code: Select all

[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% cat runner.py
import time


def main():
    while True:
        print(time.time())
        time.sleep(1)


if __name__ == "__main__":
    main()


Code: Select all

[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% cat executor.py
from subprocess import Popen, PIPE


def main():
    print("Starting runner...")
    p = Popen(["/usr/bin/python3", "-u", "runner.py"], stdout=PIPE, bufsize=1)  # Note the "-u" flag...
    print("Runner started! Runner output follows...")

    with p.stdout:
        for line in iter(p.stdout.readline, b''):
            print(line.strip())


if __name__ == "__main__":
    main()


And the current time from runner.py will be displayed in the executor.py script:

Code: Select all

[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% python executor.py
Starting runner...
Runner started! Runner output follows...
1610418164.4193301
1610418165.423
1610418166.427841
1610418167.431348
1610418168.433428
^CTraceback (most recent call last):
  File "runner.py", line 11, in <module>
Traceback (most recent call last):
  File "executor.py", line 15, in <module>
    main()
  File "executor.py", line 10, in main
    for line in iter(p.stdout.readline, b''):
KeyboardInterrupt
    main()
  File "runner.py", line 7, in main
    time.sleep(1)
KeyboardInterrupt


I have a python script invoked with Python 2.7.15 spawning another python script using Python 3.8.5, so it matches your scenario.

Code: Select all

[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% python --version
Python 2.7.15
[Aarons-MacBook-Pro:~/experiments/python_proc_comms] Aaron% python3 --version
Python 3.8.5


I'll keep that in mind. Right now I'm testing some changes the Camect team sent me using the direct websocket connection. If I can get that to work I won't need the subprocess method.