iMessage Plugin - - Big additions Wit.ai Processing

Posted on
Mon Nov 12, 2018 12:07 am
GlennNZ offline
Posts: 523
Joined: Dec 07, 2014
Location: Central Coast, Australia

iMessage Plugin - - Big additions Wit.ai Processing

Beta 0.2.2

New gitHub Branch:
https://github.com/Ghawken/iMessagePlugin/tree/wit.ai

Beta Functions: From Version 0.2.2

As a mechanism of controlling all indigo devices and requesting information with minimal setup I have turned to a free service wit.ai


Image

The aim of this internet service is to process any text messages you might send and enable the Plugin to reply, recognising what you want to do but also which device to apply this to.

e.g
Aiming to bring to life.. the following with minimal setup.

- Turn off pool Pump.
- Dim the living room light to 20%
- Set Living Room light to 100% brightness
- What is the temperature in Maxs Room?
- Tell me a joke?
- What is the temperature of the pool?
- Can you help with some advice?
- How hot is the Living Room?
- What is Glenn's Location?

Any Natural language can be trained to have the desired result within Indigo. As the Beta progresses can add other example sentences/structures and samples depending on own usage.

Basically Wit.ai - recognises message:
intent, currently device_action, insult, yes_no_decision, joke, dim_set, temperature, advice
& entities such as device_name (which is Indigo Device), plus others such as number/sentiment/on_off etc.
This parsed result is sent back to the plugin to action. (very quickly)

So if message is sent 'Please Indigo could you be a dear and turn on the living room light?'
The (Plugin after suitable wit.ai training/samples) should recognise intent: device_action, device_name:'Living Room light', and on_off = on.
The plugin parses the reply and then Turns on living room light for example.

Within Wit.ai you can train any sentence structure to the correct intent and result. The aim is for the plugin to probably do this at Wit.ai app creation which is what currently happens, obviously others may speak in a different manner. :roll: and I can add any samples/sentence structures wished.

It also enables speech->text via iMessage audioMessage, and a few fun extras perhaps I got carried away with.

Consider this a beta - BUT if disabled wit.ai in PluginConfig - rest of plugin unchanged and none of this is run.

NB:
The wit.ai processing of the sent message only occurs after any trigger check.
So if you have a trigger action set for the message 'Gate', this will work unchanged.
After receiving that message the Plugin will active your Trigger and end any thinking about the message.
If you sent 'Indigo please open gate' then that message will be sent to wit.ai and hopefully the same result will occur.
(the benefit here is natural language, and no trigger setup is needed)


Requirements:

Need your own Wit.Ai API Key (free)
We each need our own as our devices are named something different
So while the NLP logic may be the same the devices referred to will be different.

Setup:

New Plugin Config Settings

Image


1. Create Wit.AI Api
- Log in to, with either github or facebook.

https://wit.ai

Image


2/ Next Go to the already existing App called 'MyFirstApp'
- Go to Settings, top Right
- Find the 'Server Access Token' and copy this in the PluginConfig

Image


Enable Use Wit.Ai Processing to use this service for processing of Messages received.


Checkbox: Use all Indigo Devices

- This will send every indigo device name to Wit.Ai to be recognised as a device down the track.
- This is fine if you have limited devices and the names of your devices make sense for control
- But if like me you have hundreds will names like (U) Computer Room Light, or PiBeacon_Pool Temperature
- These names make sending a text message 'what is the PBeacon_Pool Temperature' will work but isn't much fun.

- I have added the ability for WitAI to recognise synonyms for the same device
- eg. device is PiBeacon_Pool Temperature, other names can be 'Pool Temp', 'Pool Temperature' or even 'Pool'

This is performed via the individual Device Notes of each Indigo Device. (once there is re-read)

On the first line of the notes (the other lines are ignored)
a :

Code: Select all
witai|Pool|Pool Temp|Pool Temperature


- Must start with witai and then Vertical bar Characters separating names
- Sorry the character is the | Vertical Bar Character

eg.

Image

- This will give different names which refer to the same device for all future commands

Coming back to the Checkbox - if this is unchecked only those devices so marked with witai| in device notes will be included.

Okay:

So ideally - give you funny named devices some better names, mark those you want to send, or send all

Then:

1. Press the Button called 'Generate Wit.Ai App' [once we are done only need to do this once]

This will send a whole lot of text based data to Wit.ai naming devices, and setting up you wit.ai device which the plugin uses
You can access this online at wit.ai if needed but really only for finetuning.

This will take a while as can only send a few devices/samples at once; so will take at least 5 if not 10 minutes to be done.
Once finished also takes a while at Wit.Ai end as well - before everything is recognised probably - probably at least 1/2 hour

If any issues or want to resend different devices following some playing - press the delete wit.ai App button and then recreate after a short pause.


Following this

- You should be able to 'turn on INDIGODEVICE'
- 'turn off INDIGODEVICE',
- 'set INDIGODEVICE brightness to 50%',
- 'What is the Temperature of INDIGODEVICE',
- 'Tell me a joke'
- 'What is IFRIENDEVICE location'
- Please give me some advice

- amongst others and does have constant learning on its side... please you can adjust from within wit.ai if there are sentences you use
(the intent in wit.ai is the most important - and should be self-explanatory)

I have also added iMsg AudioMessage speech to text via wit.ai - which works okay within limits of speech recognition.
From within iMSG press and hole down microphone - say command and viola!

Image



Fine Tuning:

- Probably more to come I suspect:
- Basically can add whatever phrase seems to make sense to its samples and can modify and build the training data which is sent when the app is created.

- From within Wit.ai - can go to Inbox and validate audio received for training purposes .

- Can also can to device_name and add any extra synomyms for this device you wish - pays to do it in Indigo Notes as well/so survives delete and recreate of the wit.ai app

Other changes:
- Add send HTTP Image - takes http/web location downloads fine and sends it to buddy.
- Few other fixes/changes


New gitHub Branch:
https://github.com/Ghawken/iMessagePlugin/tree/wit.ai
(Clone or Download)
Last edited by GlennNZ on Fri Dec 14, 2018 4:23 pm, edited 5 times in total.

Posted on
Mon Nov 12, 2018 11:46 am
siclark offline
Posts: 836
Joined: Jun 13, 2017
Location: UK

Re: iMessage Plugin - Beta 0.2.2 - Big additions

Wow. Yet more things I want to integrate but just don't have time for!! Can't wait to see what this can do.

However it's got me thinking, could this be used to do build our own version of Alexa? I'm thinking of better 2 way conversation and setting variables etc, maybe with a rPi and mic and possibly speaker?

Posted on
Mon Nov 12, 2018 12:10 pm
mat offline
Posts: 742
Joined: Nov 25, 2010
Location: Cambridgeshire - UK

Re: iMessage Plugin - Beta 0.2.2 - Big additions

Wow again, you 've been busy! Looks great. Will test soon.

Late 2018 mini 10.14

Posted on
Mon Nov 12, 2018 2:17 pm
GlennNZ offline
Posts: 523
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: iMessage Plugin - Beta 0.2.2 - Big additions

siclark wrote:
However it's got me thinking, could this be used to do build our own version of Alexa? I'm thinking of better 2 way conversation and setting variables etc, maybe with a rPi and mic and possibly speaker?


Basically that is what it is - or certainly possible to head towards eventually

The voice recognition is only so/so - fine for turn on living room after some correction in the wit.ai website. But not good enough to listen forever for wake word. Although there are some libraries for local voice recognition of the wake word.... but .... hmmm.....

Posted on
Tue Nov 13, 2018 7:34 am
roussell offline
User avatar
Posts: 1085
Joined: Aug 18, 2008
Location: Alabama

Re: iMessage Plugin - Beta 0.2.2 - Big additions

If you’re going down the local ai vr route, take a look at https://snips.ai, It’s fun to play with. The biggest problem these days is the availability of a low cost far-field microphone array with DSP to hear and separate the voices from the background noise. If only the dots were hackable, crap even if Amazon sold a “maker” version for more $ it would be worth it

Terry


Sent from my iPhone using Tapatalk

Posted on
Tue Nov 13, 2018 11:49 am
mundmc offline
User avatar
Posts: 432
Joined: Sep 14, 2012

Re: iMessage Plugin - Beta 0.2.2 - Big additions

Holy cow, this looks awesome.

Terry- have you played around with Alexa skills yet? I’m just starting to tinker. Getting open ended info from Alexa is a pain. I’ve explored the whole “literal and custom” slot approach, and I’m still working on just scraping the last heard command (getting closer).

Posted on
Tue Nov 13, 2018 3:53 pm
siclark offline
Posts: 836
Joined: Jun 13, 2017
Location: UK

Re: iMessage Plugin - Beta 0.2.2 - Big additions

Awesome indeed. Seriously thinking of a Christmas project!!

Posted on
Tue Nov 13, 2018 4:29 pm
siclark offline
Posts: 836
Joined: Jun 13, 2017
Location: UK

Re: iMessage Plugin - Beta 0.2.2 - Big additions

So just $190 for case, mic array, and pi or equivalent to run this on!!

https://www.amazon.com/ReSpeaker-Case-v ... B075MC125Q

Certainly not cheap Vs $30-50 for a dot, especially as most of us have many, but local voice control is appealing. Not that I would have time to set this up anyway.

Posted on
Tue Nov 13, 2018 4:37 pm
siclark offline
Posts: 836
Joined: Jun 13, 2017
Location: UK

Re: iMessage Plugin - Beta 0.2.2 - Big additions

PS sorry Glenn for hijacking your thread.

Posted on
Thu Nov 15, 2018 12:39 am
GlennNZ offline
Posts: 523
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: iMessage Plugin - Beta 0.2.2 - Big additions

Beta 0.2.5 Changes

https://github.com/Ghawken/iMessagePlugin/releases

Recap:
Image

Uses wit.ai online service to parse both device names and also sentence intent.
Should with minimum of input parse the following without problem:

    - Turn off pool Pump.
    - Dim the living room light to 20%
    - Set Living Room light to 100% brightness
    - What is the temperature in Maxs Room?
    - Tell me a joke?
    - What is the temperature of the pool?
    - Can you help with some advice?
    - How hot is the Living Room?
    - What is Glenn's Location?


Change Log:
0.2.4

New PluginConfig options to update online Wit.ai app:

Image

Changes to PluginConfig - hide the scary wit.ai stuff if not using
Allow Wit.ai device to be updated with app - update button
Update Images Documentation

0.2.3
- Add Specific Buddy Command Received Trigger:


Image


Here: As Pre-release:
https://github.com/Ghawken/iMessagePlugin/releases

Glenn

Posted on
Fri Dec 07, 2018 10:11 pm
dnomode offline
Posts: 354
Joined: Apr 12, 2008
Location: North Georgia

Re: iMessage Plugin - Beta 0.2.5 - Big additions

I tried to setup wit.ai today and I can't get it to work.

I've tried to setup 3 devices. My garage doors and my office light.

I've setup the devices like this picture.
Screenshot 2018-12-07 22.57.07.png
Screenshot 2018-12-07 22.57.07.png (189.21 KiB) Viewed 475 times


I tried sending some text however my devices don't react as planned. I just get random texts back.
Screenshot 2018-12-07 23.06.28.png
Screenshot 2018-12-07 23.06.28.png (64.72 KiB) Viewed 475 times


I've also received a 404 error, device not found and timeout error. Please see below.

Any ideas how I can fix this.

Thanks
Edmond

Code: Select all
Dec 7, 2018 at 10:46:43 PM
   iMessage Plugin Error           New Access Token Equals:VWNN864SF52AJA5BHPTAVKULQD43MLYL
Dec 7, 2018 at 10:49:16 PM
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           Indigo iMessage wit.ai Application Created Successfully.

Dec 7, 2018 at 10:49:53 PM
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547

Dec 7, 2018 at 10:53:26 PM
   iMessage Plugin Error           Wit responded with status: 404 (Not Found)
   iMessage Plugin Error           Wit responded with an error: Entity 'device_name' not found
   Error (client)                  timeout waiting for plugin response from com.GlennNZ.indigoplugin.iMessage for request UiAction
   iMessage Plugin                 Imessage Plugin:  wit.Ai Device successfully updated.

Dec 7, 2018 at 10:54:35 PM
   iMessage Plugin Error           Wit responded with status: 404 (Not Found)
   iMessage Plugin Error           Wit responded with an error: Entity 'device_name' not found
   Error (client)                  timeout waiting for plugin response from com.GlennNZ.indigoplugin.iMessage for request UiAction
   iMessage Plugin                 Imessage Plugin:  wit.Ai Device successfully updated.

Dec 7, 2018 at 10:55:08 PM
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547

Dec 7, 2018 at 10:55:34 PM
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547
   iMessage Plugin Error           iMessage Received from Buddy(s):  Buddy(s) Handle Below:
   iMessage Plugin Error           +1879561547

Posted on
Fri Dec 07, 2018 10:37 pm
GlennNZ offline
Posts: 523
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: iMessage Plugin - Beta 0.2.5 - Big additions

If still there turn debug logging on and PM me it - I can have a look now.

Edit:

Need to turn off the show Buddies in the Plugin Config - if you know the buddy handle don't need anymore.
That is most of the 'error' / Logs

Would presume you haven't generated wit.ai App - this needs to be done first before anything else.
Indigo UI UI may may timeout when doing this - can ignore.

And can check on wit.ai website that your indigo device created and should show names for all devices etc.

G

Posted on
Fri Dec 07, 2018 10:42 pm
dnomode offline
Posts: 354
Joined: Apr 12, 2008
Location: North Georgia

Re: iMessage Plugin - Beta 0.2.5 - Big additions

just sent you PM


Sent from my iPhone using Tapatalk

Posted on
Fri Dec 07, 2018 10:48 pm
dnomode offline
Posts: 354
Joined: Apr 12, 2008
Location: North Georgia

Re: iMessage Plugin - Beta 0.2.5 - Big additions

OK, I turned off Show Buddy.

The messages stated that Imessage Plugin: wit.Ai Device successfully updated.

However I don't see any devices on Wit.ai website. Where would they be?

FYI: I do see my messages in the inbox

What you think?

Posted on
Fri Dec 07, 2018 10:51 pm
GlennNZ offline
Posts: 523
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: iMessage Plugin - Beta 0.2.5 - Big additions

Tah
Hard to know what is completely going on from that PM log - sending to wit.ai - just doesn't seem to have a app to process the commands.
So because of this - getting somewhat random replies. NB has to be one single message together, won't process multiple lines.

But have you generated a Wit.Ai App? Doesn't seem like you have which is issue.
Shouldn't need to update - just generate once. Can update if add new devices..

(If you can debug log all below and PM that would be ideal)

I would go to iMessage Plugin Config

Delete Wit.Ai App Button

SAVE, and Return to PluginConfig for good measure

Enter Access Token - this is the main server access token for only Wit.Ai app you should have.
MyFirstApp - Here from above:
Image


Then Generate Wit.Ai App button again

Probably will time out - takes 10-15 minutes to send everyting.
(although in your case with only 3 devices probably not that long)

Save
Takes a bit of time for Wit.AI processing to be complete
But if go to wit.ai - should see app there and be able to look at details of it.

G

Who is online

Users browsing this forum: No registered users and 0 guests