The Python plugin API and "Twisted"

Will the plugin framework impose its own event loop, or will it be possible for me to use Twisted? That would be really nice as it would let me easily throw in a web server, email client, or any of the other things that have already been developed for it.

Basically Twisted looks like the Python equivalent of POE, so I might use it as an intermediate step just to learn Python and get my stuff ported before the plugin API comes along.

[jay (support)]

We're thinking of using asyncore in most of our examples because it seems lighter-weight, but we did some rough testing and got a simple twisted-based plugin running that listens on a port and stuffs what it sees into the event log. It appears to work well.

Mostly, we believe that putting together quickly a prototype using twisted demonstrates to us that the framework we've put in place is flexible enough to handle a lot of different scenarios. We're working feverishly on getting it to a stable alpha state...

OK, so just to be clear, the _plugin_ code is responsible for the event loop?

That sounds like "one persistent process per plugin", which is good, but I am confused with respect to the architecture you hinted at earlier, where timers and threading would be done in the plug-in host. For example, I could imagine timer facilities being provided through an API by the host, even if the plugin is his own process/thread, but that's another thing you already get with Twisted.

I will take even an _unstable_ alpha when you have it, or just some example plugin code to start thinking about.

[matt (support)]

The plugin host process embeds python, and calls into the plugin's python class to create the plugin instance. It then, when IndigoServer (or Cocoa client) initiates actions (ex: plugin menu item being selected, a plugin defined action needs to be executed, etc.), calls out to the python plugin instance defined callback methods. Example callbacks might change a plugin preference, send a command to a serial queue, generate a report, change the state of a device, etc.

One such optional callback function is runConcurrently(). If the plugin defines that, then the host automatically creates a new thread and calls that function after the plugin is created. The plugin never returns from that function (unless the host tells it to shutdown), so from there the plugin can:

• Loop doing periodic processing with a sleep call (NOAA plugin, iTunes sync plugin, etc.)
• Use asynccore to handle network and serial I/O
• Use Twisted to handle all events (call reactor.run())

The other python callback functions (say, for an IndigoServer triggered action) can communicate with the main runConcurrently() thread however they want, via a queue, asyncore or Twisted. Twisted isn't thread safe, so inside python callbacks you have to defer the function call, but twisted makes that easy with the reactor.callFromThread() function.

So I think to answer your question: the plugin is responsible for its own event loop but there are callbacks into the plugin as well. And it is one persistent process (we call it the Indigo Host client) per plugin. The Indigo Host is the master of the python plugin instance, but will let the plugin pretty much run on its own if needed via the runConcurrently() function. IndigoServer will manage launching all of these host processes and stopping them if the plugin is disabled.

Everyone please note these are technical details, and we'll be providing lots of documentation and examples. It will be much easier to use than the above might sound.

[jay (support)]

And there are GAPING holes yet to be filled - so just because the above may sound great, there is a ton of work still to be done before we can even start doing alpha testing. We appreciate everyone's patience and understand that there's a pretty good level of interest here. We think this API is gonna be extremely cool - we just need some more time to get it put together.

You guys are brilliant! I don't know what else to ask for.