Voice control of Indigo - anyone interested?

Posted on
Thu Feb 27, 2014 2:16 pm
dstrickler offline
User avatar
Posts: 306
Joined: Oct 08, 2010
Location: Boston, MA

Re: Voice control of Indigo - anyone interested?

So, I finally got around to testing out your app, hooked it into my NLP* system, and I am stunned - full NLP voice control over my house. Ironically, I'm speechless! With your Applecsript code you just saved me *weeks* of programming to test if a prototype would work. THANK YOU!!!

I'm not at a point where the code understands anything except my house, but I'll work on that soon, and post a demo when it's ready.

-- Dave

* "NLP" is Natural Language Processing; the ability to translate a sentence into machine instructions.

Dave

Posted on
Fri Feb 28, 2014 7:06 am
mundmc offline
Posts: 198
Joined: Sep 14, 2012

Re: Voice control of Indigo - anyone interested?

:D Very glad to be helpful!
Keep me posted!

Posted on
Fri May 09, 2014 8:12 am
mundmc offline
Posts: 198
Joined: Sep 14, 2012

Re: Voice control of Indigo - anyone interested?

Updates???

Posted on
Thu May 15, 2014 12:00 pm
dstrickler offline
User avatar
Posts: 306
Joined: Oct 08, 2010
Location: Boston, MA

Re: Voice control of Indigo - anyone interested?

Zippo I'm afraid.

I've been busy with a work project and haven't had any time to play with things. When work dies down a bit, I will dig into it again.

Dave

Posted on
Sat May 24, 2014 10:44 am
mundmc offline
Posts: 198
Joined: Sep 14, 2012

Re: Voice control of Indigo - anyone interested?

I'm on the verge of writing something similar in python, but I was hoping you would beat me to the punch (my code tends to work, but it's anything but elegant as I learn as I go along).

I'm playing with something that MIGHT be Natural Language Processing (NLP):
1) Common word-combinations (e.g. "turn on", "turn up", "change to" ) are swapped for single words (e.g. "activate", "raise", "select")
2) It keeps a separate data file with known words, grammatical usage (e.g. noun or verb), as well as likely context as follows:
-----> E.g. "Turn on Netflix" would be interpreted as the following
-----> "Turn on" could mean "activate" or "select"
-----> "Netflix" (in my case) has a context field of AppleTV
-----> Since all of my current house-stuff states are kept in variables (e.g. Indigo knows what's on or off, what's selected on the receiver, etc), it interprets this command as follows: mundmc wants the receiver, tv, and apple tv to be on, and he wants appletv to select netflix (this is all done via an IP2IR box and the Cynical Cache plugin, which is wonderful)
-----> this is parsed to the string "select AppleTV subroutine AppleTV select Netflix" (subroutines are context sensitive, in this case AppletTV)
-----> text after "subroutine" is split off into it's own string and executed after a pause
-----> "select" is a known word, indicating the stereo will be changing inputs; "AppleTV" is also a known word, indicating which inputs will be selected (in my case, TV on "HDMI1", receiver on "AUX")
-----> pause
-----> the AppleTV is sent a series of commands to bring it to the home screen (essentially, the "menu" command about 10 times, rapidly), then, Netflix is selected (in my case, the commands "right" "right" "down" "down" "select")

3) the challenge: I would like the code to learn commonly used words, as well as fill-in implied words
-----> ex.: if my wife shouts "AppleTV" (because she has no regard for syntax or semantics), there is no apparent verb (e.g. "select" or "activate") and the code defaults to the verb "select AppleTV"
-----> I would like to have the python code learn new, less common phrases, and apply syntactic rules to them, ex.: "put on AppleTV" could prompt the user for commonly used verbs if the phrase "put on" is not recognized: Indigo asks "do you want activate apple TV", and waits for a response of "yes"; a "yes" response stores "put on" as a synonym for "activate"
-----> the last part gets complicated quickly, as it involves speech output and recognition WITHIN the code that is already running; this is where I am uncertain how to proceed; possibly by having python tell Indigo to talk AND listen within one script

(sigh)

Thoughts appreciated on a more elegant way to do this, as I'm sure there is one

Posted on
Tue May 27, 2014 10:56 am
dstrickler offline
User avatar
Posts: 306
Joined: Oct 08, 2010
Location: Boston, MA

Re: Voice control of Indigo - anyone interested?

As far as your code using NLP, it is, in the strict sense. It's english, and you're parsing it ;-) Is it "state of the art"? No, but it's clever, and I applaud you for trying it out - it's hard to do NLP at any level.

Your real issues (and I'm guessing here), are stemming from #3, which is the base problem that NLP solves - "turn on the lights" should be the same as "turn the lights on". Sounds simple, but the permutations on even a simple level are mind boggling to account for.

The code I have working uses a professional NLP engine for parsing the grammar into base commands, so that "turn on the kitchen lights" translates to
Action: Turn On
Target: Lights
Location: Kitchen
So you don't need to worry about all the NLP processing's (trust me - let someone else deal with this headache). Once you have the english parsed into base commands, the actions are pretty easy.

Now, that all being said, I'll tell you what stalled my project (other than lack of time to work on it) - verbal dialog.

The first problem is pretty easy for me. After a command is issued like "turn on the lights", I can see that the location is missing, and send back text saying "Which lights do you mean?", and then parse that command. Pretty standard when you are using a keyboard, and thus text only.

But, the second is making this verbal dialog. Your solution/hack of using Siri with the Notes feature is really clever (applause for figuring that out), but it lacks the next steps for Siri to come back and say "Which lights do you mean?"

I think the solution needs to be an phone-based app that can take dictation and send it up to be parsed (I have hacked this together) and then something that converts text from the server from "Which lights do you mean?" into spoken (verbal) dialog. After all, that's where we want, right - interactive dialog?

So my long post here is to say that unless we attack this from a bi-directional phone app, I think we will be disappointed in the end result - it just won't do enough.

I don't know how to code iPhone apps (but under enough pressure I could learn), but on the horizon is something interesting. There are rumors floating around that Apple will announce some Home Automation hooks to Siri at the WWDC on June 2nd.

Now I hate waiting for anything like this, but since it's very soon, I think we should wait to see what they have in store. If it's good, I can re-up my developers license, and dive in. If not, we can see if we can get someone (or me) to develop a phone app to be how you interact with Indigo.

Either way, I think a solution is possible, and it could be very impressive.

Comments?

Dave

Posted on
Sun Jun 01, 2014 8:47 am
mundmc offline
Posts: 198
Joined: Sep 14, 2012

Re: Voice control of Indigo - anyone interested?

I think I understand where you are coming from, insofar as having a "back and forth". The new apple home automation stuff, unfortunately, sounds more like creating a standard than opening anything up: http://9to5mac.com/2014/05/29/apples-ho ... e-program/ .

Similarly unfortunate, there aren't any results on Siri Proxy yet either: https://github.com/plamoni/SiriProxy/issues/542

I am legitimately considering investigating google phones just for this functionality.

I very much appreciate your response!

Posted on
Sun Jun 01, 2014 6:14 pm
dstrickler offline
User avatar
Posts: 306
Joined: Oct 08, 2010
Location: Boston, MA

Re: Voice control of Indigo - anyone interested?

Well, I'll hold out to see what they say in WWDC, but no, I'm not holding out for a miracle solution ;-)

I may (about 50/50 at this point) have some free time coming up to work on this. I'll know for sure in about 10-14 days.

Dave

Posted on
Mon Jun 02, 2014 1:06 pm
dstrickler offline
User avatar
Posts: 306
Joined: Oct 08, 2010
Location: Boston, MA

Re: Voice control of Indigo - anyone interested?

"HomeKit" has been announced as predicted. http://www.tuaw.com/2014/06/02/wwdc-201 ... r-one-app/

Looks like there is integration with Siri ("Siri, close the garage door") which solves a huge problem on my part. But, let's not forget this is just an announcement. I have no idea what the code looks like, what the API is, or what it will entail. These are giant unknowns.

That being said, you may want to hold off on the Android switch for a few months ;-)

I'll be following HomeKit closely, and will keep you posted on this thread.

Dave

Posted on
Sat Sep 13, 2014 12:50 pm
mundmc offline
Posts: 198
Joined: Sep 14, 2012

Re: Voice control of Indigo - anyone interested?

U2 was cool and all, but not much announced on home automation at WWDC. I'm looking around for less publicized Home Kit news.

Posted on
Sun Sep 14, 2014 8:41 am
pvrfan offline
Posts: 53
Joined: Apr 19, 2011
Location: Near Toronto, Canada

Re: Voice control of Indigo - anyone interested?

You have to register for an Apple Developer account (free) but then you can access the details of the software API:

https://developer.apple.com/library/pre ... index.html

There is also a video of the WWDC session on HomeKit that gives an overview of how the software side works (~40 minutes). I'm too lazy to look up a link to it right now. Unfortunately it doesn't answer any questions about the hardware side. One has to qualify for the "Made for iPhone" program in order to get the hardware info.

A couple of manufacturers have already announced HomeKit compatible products:

http://www.macrumors.com/2014/09/02/elg ... mart-bulb/

http://recode.net/2014/08/27/one-of-the ... ice-ships/

Craig

Posted on
Sat Dec 06, 2014 6:19 pm
vasisht offline
Posts: 36
Joined: Dec 08, 2012

Re: Voice control of Indigo - anyone interested?

I'm working on a natural language processor myself - out of curiosity, what language did you use to program this?

Posted on
Wed Jan 21, 2015 10:30 pm
whmoorejr offline
User avatar
Posts: 214
Joined: Jan 15, 2013
Location: Houston, TX

Re: Voice control of Indigo - anyone interested?

If Apple's decision to sidestep a software bridge.... (shunning the die hard MAC automation folks to pander to the masses) bothers you too....

Add to my thread at Apple.... or start your own grievance there..... either way.... let the weenies in cupertino know how you feel.

https://discussions.apple.com/message/27523352#27523352

Who is online

Users browsing this forum: No registered users and 1 guest