ChatGPT powered Siri

Posted on
Sat Mar 04, 2023 5:30 pm
anyone offline
User avatar
Posts: 125
Joined: Apr 03, 2011
Location: Norfolk, VA

Re: ChatGPT powered Siri

Similiar experience here, with most of my frustration with the unsuable formating of the returned JSON.
Its was difficult to know if it was a problem with something I just did to my prompt or not, so it slowed my progress.

The docs for gpt-3.5-turbo say its the 'best model for many non-chat use cases' at a reduce cost. I played with it as well, for me it seems to perform much worse than text-davinci-003. Again, not sure if that's my prompt or not.

Sending the entire prompt each time did appear to be the only option. I also experience it making up rooms, so I'm giving it rooms now. Which allows you to say turn off the X in the Family Room or all of the lights in the X room.

I put in, took, out then put some verbiage about who I am so it would know if I said turn off 'my whatever' it could from the device name know what I was talking about in any room. Then...I thought about, what about my partners stuff. Then....what about....this quirk in my home. I should explain that.

So with a growing prompt, I looked into pricing. Currently my prompt is 1,977 tokens or 6714 characters.
(https://platform.openai.com/tokenizer)

The gpt-3.5-turbo model is priced at $0.002 per 1k tokens. (https://openai.com/blog/introducing-cha ... isper-apis)

Someone double check me on ALL of this, thats $0.004 per request. So if I make 50 requests per day over 30 days, that's 1500 requests @ $0.004 per request which is $6 USD/mo rounding up. lol. Interestingly this is a subject ChatGTP is not so well versed in. However, ChatGTP did explain the ChatGTP Plus users do not get any free tokens and pricing for the API is totally separate.

Do you think you can work thru the unreadable JSONs or are just accepting you'll have to toss some responses?

Posted on
Sat Mar 04, 2023 5:39 pm
siclark offline
Posts: 1960
Joined: Jun 13, 2017
Location: UK

Re: ChatGPT powered Siri

Do you have examples of the bad JSON?

Yes I think you do have to tell it what rooms you have and what the functionality within each is.

That was as they original example and I'm sure he did a lot of trial and error to get it working well

Posted on
Sat Mar 04, 2023 6:19 pm
GlennNZ offline
User avatar
Posts: 1555
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: ChatGPT powered Siri

The bad JSON is a pain:
‘Response : ‘ then {— then correct Json
or
quotes wrong
or
no JSON at all - just a text response

Working around most of it - but it is annoying…. Changing the system content doesn’t seem to have much impact.

Toying with testing other replies - comma separated etc, suspect will probably be same outcome.

I’m finding gpt3.5-turbo - more consistent. Not necessarily more accurate - just more consistently in what it gets wrong and hence work-around-able.

Also odd replies - with multiple JSON’s returned to simple request.

If you give it a simple request to a Device which you have told it about, it works fine.

“Turn off Lights” - and see what reply you get…

The plugin programmatically gives it all the device names, synonyms for that device name and the device ID with every request (as you have to). I haven’t setup Rooms as yet.


Sent from my iPad using Tapatalk

Posted on
Sat Mar 04, 2023 7:02 pm
GlennNZ offline
User avatar
Posts: 1555
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: ChatGPT powered Siri

Ok: This is getting really annoying - nothing we hate more than different replies for the same input!

To ChatGPT: (There is a set Device with name Dimmer, and DeviceID specified
"Set Dimmer to 50%"

One reply: (same system setup sent each time)
Code: Select all
{
    "action": "command",
    "location": "clarify",
    "target": "Example Dimmer",
    "ID": null,
    "value": 50,
    "comment": "Sure, I can set the dimmer to 50%."
}


Next reply:
Code: Select all
{
  "action": "command",
  "location": "unknown",
  "target": "Example Dimmer",
  "ID": 1323480013,
  "value": "50%",
  "comment": "Dimmer has been set to 50%"
}


3rd:
Code: Select all
{
    "action": "command",
    "location": "unknown",
    "target": "Example Dimmer",
    "ID": 1323480013,
    "value": 50,
    "comment": "Dimmer set to 50%."
}


And so on...
These might seem similar but -- you note first one doesn't include Device ID which is somewhat fatal to any action.
2nd one - includes Device ID - but value to set is now a string including 50% - again pretty fatal.
3rd - includes deviceId, and value in Int, target name is now the synomyn for some reason....

The natural language stuff is working really well, the device selection stuff -- Hmmm..
Not sure am going to spend to much more time on this ...

Posted on
Sun Mar 05, 2023 12:36 am
GlennNZ offline
User avatar
Posts: 1555
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: ChatGPT powered Siri

Okay - some prelim support here, which may not suit the device Control flavour of this thread...
https://github.com/Ghawken/iMessagePlugin/tree/Python3

Overall not that useful in my testing regarding device Control - works okay, but so does Alexa and HomeKit really.
Pausing this for a bit because think Sessions will come to openAI API - which will improve functionality a lot.
& probably a lot of finetuning to be done to the setup system info and details given.

What is working very well is the general chatbot, answering questions, providing reasonable chatGPT based advice on a number of topics.
Am running this across the family now - basically they text Indigo - commands which still trigger, Gate, Lights off etc. - but if not a command can be question which gets a reasonable reply..

The API is very cheap - so far in hundreds and hundreds of test queries (some large essay's) I have spent $1.75, out of $18 dollar credit that everyone gets when signup...

Glenn

Posted on
Sun Mar 05, 2023 2:36 am
siclark offline
Posts: 1960
Joined: Jun 13, 2017
Location: UK

Re: ChatGPT powered Siri

GlennNZ wrote:
Ok: This is getting really annoying - nothing we hate more than different replies for the same input!

To ChatGPT: (There is a set Device with name Dimmer, and DeviceID specified
"Set Dimmer to 50%"

One reply: (same system setup sent each time)
Code: Select all
{
    "action": "command",
    "location": "clarify",
    "target": "Example Dimmer",
    "ID": null,
    "value": 50,
    "comment": "Sure, I can set the dimmer to 50%."
}


Next reply:
Code: Select all
{
  "action": "command",
  "location": "unknown",
  "target": "Example Dimmer",
  "ID": 1323480013,
  "value": "50%",
  "comment": "Dimmer has been set to 50%"
}


3rd:
Code: Select all
{
    "action": "command",
    "location": "unknown",
    "target": "Example Dimmer",
    "ID": 1323480013,
    "value": 50,
    "comment": "Dimmer set to 50%."
}


And so on...
These might seem similar but -- you note first one doesn't include Device ID which is somewhat fatal to any action.
2nd one - includes Device ID - but value to set is now a string including 50% - again pretty fatal.
3rd - includes deviceId, and value in Int, target name is now the synomyn for some reason....

The natural language stuff is working really well, the device selection stuff -- Hmmm..
Not sure am going to spend to much more time on this ...
Interesting. How is your request to ChatGPT setup? Strange that it's not asking for clarification for which dimmer and happily creating an example one or supplying blank.
It's also a strange request, is this a light dimmer?
What if you ask to set the light to 50%?
How have you explained that it can control dimmers? The original text explained it could dim lights so it knows a light can have 50% value, it sounds like it's guessing at the dimmer code.

Posted on
Sun Mar 05, 2023 5:03 am
GlennNZ offline
User avatar
Posts: 1555
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: ChatGPT powered Siri

Yeah - pretty sure those were given, Dimmer is a dummy dimmer device.

But agree it will depend on the setup, but even so is a bit flakey. Until the api has the ability to use saved sessions which remember previous conversations - having to send everything with every request is going to be an issue, both hitting the limits, cost, and amount of devices/rooms can be setup..

The fam are enjoying the ‘Indigo’ written Odes from iMsg chatGPT though!
“Write me a ode about 2 merry programmers Matt and Jay from Indigodomo smart home software”. Hang on, with just swap apps… and

Absolutely! Here's an ode to Matt and Jay from Indigodomo smart home software:

Oh, merry programmers of Indigodomo fame,
Matt and Jay, who bring smart homes to claim!
With diligent hands and a sharp intellect,
They pave the way to a connected prospect.

Their minds alight with creativity's flare,
They craft and code with a focused care.
From dimming lights to shades that sway,
Their skills bring your home into the future's fray.

And with each update and every patch,
Matt and Jay bring life to your humble match.
For smart homes are but a dream without,
The brightest minds to make our home's intelligence sprout.

So here's to thee, oh programmers divine,
Who made our homes a place to align.
Your craft and care in every line,
Bring forth the future, both yours and mine.”


Sent from my iPad using Tapatalk

Posted on
Sun Mar 05, 2023 5:25 am
siclark offline
Posts: 1960
Joined: Jun 13, 2017
Location: UK

Re: ChatGPT powered Siri

Ok I just ran 10 requests and yes one of them had. % sign in the value.

Think that could be stopped with an explicit instruction that it can be number only or on/off etc.

I am getting responses for rooms that don't exist so beef to be explicit there too.

Yes better when we can continue conversions.

Or what about training the model for our use case?

https://platform.openai.com/docs/guides/fine-tuning

Posted on
Sun Mar 05, 2023 2:10 pm
GlennNZ offline
User avatar
Posts: 1555
Joined: Dec 07, 2014
Location: Central Coast, Australia

ChatGPT powered Siri

Yeah - it’s inconsistency that is the issue. I have it marking up Device ID and names fairly regularly - they just get rejected.

I did see the fine tuning - but didn’t look at it in detail. Seems to also cost x9 as much? when then later used? Thought this was probably in the land of over complicating…

[edit]
Had a look and you can send hundreds of example questions with expected outputs to the fine-tuning aspect of GPT-3. This would be straight forward to do, and currently do so for wit.ai. chatGPT API will que it and take some hours to be done, it would seem. Once finished you then use this fine tuned model (seems these are user based - so can’t be used by all - but need to double check this) for all requests.

Given experience with training of wit.ai - suspect this will lead to a very good response rate. Problem will be the time involved in training aspect and the ultimate cost (>x9 Davinci, or x90 cost of gpt-turbo!). Not a wide spread option I would suspect.

Glenn


Sent from my iPad using Tapatalk
Last edited by GlennNZ on Sun Mar 05, 2023 4:36 pm, edited 1 time in total.

Posted on
Sun Mar 05, 2023 4:07 pm
siclark offline
Posts: 1960
Joined: Jun 13, 2017
Location: UK

Re: ChatGPT powered Siri

Agreed to all above. Would save on not using so many tokens per request as they would be small. Much smaller than current.

But will need some more time invested than it immediately appeared in the video.

Posted on
Mon Mar 06, 2023 12:25 am
GlennNZ offline
User avatar
Posts: 1555
Joined: Dec 07, 2014
Location: Central Coast, Australia

Re: ChatGPT powered Siri

Yeah.
The training will make a huge difference - and as you note will lead to smaller repeated requests. In Imsg plugin with Device Control disabled, because no need for massive background. I’m sending the prior conversation history (max 3000 tokens) with every request - enables a far better chat / conversation experience
eg. Give me another example. - Type requests are fine, with chatGPT then referencing earlier points and expanding replies as needed.

I also find it’s long replies and good summaries are far better suited to text messages rather than voice - too long to listen to…

The trained data davinci model - will end up costing 12 cents for a 1000tokens, if using Davinci. Could use earlier models which for device control probably would work as well - however would loose the other aspects.

Model Usage
Ada $0.0016 / 1K tokens
Babbage $0.0024 / 1K tokens
Curie $0.0120 / 1K tokens
Davinci $0.1200 / 1K tokens

Might be possible to use cheap model to classify requests into device control or other. I would guess, without testing, that would be excellent at this. Probably would need fine tuning still - although that is exponentially cheaper as above table. And then could re-send the a chatty/information/answer type request to appropriate model. This would minimise traffic to the fine tuned expensive model, and allow use for other replies to gpt-turbo-3.5 model…. Would slow down replies a bit though.


Sent from my iPad using Tapatalk

Posted on
Tue Mar 07, 2023 9:43 pm
siclark offline
Posts: 1960
Joined: Jun 13, 2017
Location: UK

Re: ChatGPT powered Siri

I’ll prefix this by saying I’m going to focus on my new plug-in for my irrigation and that I’ve not looked into this except for a quick look at the shortcuts and realising it’s more complicated than I can understand, but

It’s now possible to continue conversations with chatgpt via shortcuts rather than starting fresh.
Not sure how it could be integrated but sharing if anyone interested.

https://github.com/Yue-Yang/ChatGPT-Siri

Posted on
Sun Mar 12, 2023 10:37 am
Different Computers online
User avatar
Posts: 2534
Joined: Jan 02, 2016
Location: East Coast

Re: ChatGPT powered Siri

it's a different LLM, but you can now run one on personal hardware.

https://simonwillison.net/2023/Mar/11/llama/

SmartThings refugee, so happy to be on Indigo. Monterey on a base M1 Mini w/Harmony Hub, Hue, DomoPad, Dynamic URL, Device Extensions, HomeKitLink, Grafana, Plex, uniFAP, Fantastic Weather, Nanoleaf, LED Simple Effects, Bond Home, Camect.

Posted on
Mon Mar 13, 2023 2:17 am
siclark offline
Posts: 1960
Joined: Jun 13, 2017
Location: UK

Re: ChatGPT powered Siri

Interesting. Thank you

Who is online

Users browsing this forum: No registered users and 4 guests