Sloooooow control pages with Indigo 2022.2.0

We are using for years now this control page address from MacBooks and iPads to reach the (local) Indigo Server:

http://192.178.188.121:8176/controlpage ... 0page%20II

At that control page are also 5 little camera feeds:
http://Userrr:!Passsword@192.178.188.52/jpg/image.jpg
(image size 165 x 120)

Works perfect!

But with the newest Indigo version these kind of pages load and refresh very slowly or don't load at all.

We've looked at the forum but we have not been able to discover the solution. Are we doing something wrong?

[matt (support)]

I just tried loading a Control Page directly (via a similar URL) and it was instantaneous even with a large CP.

Can you duplicate the CP and delete the camera feeds (I presume those are refreshing image controls??) to see if that is what is causing it to load slowly or fail to load?

And are there any errors in the Event Log?

Thanks for the feedback.

"Can you duplicate the CP and delete the camera feeds (I presume those are refreshing image controls??) to see if that is what is causing it to load slowly or fail to load?"

Yes, those are refreshing Image URLs. But removing the camera feeds didn't help either.
When we select this control page at the local Indigo program at the Indigo Server and select "Show in Browser", it takes ~ 10 seconds to load/show this control page (with 420 items).
But also with other simple control pages it takes a lot more seconds to load as before the update and with a MacBook this control page won't load anymore.
There are no errors in the Event log.

Further on I see that the name of the control page at that moment is changed to an address with an ID?

Has this something todo with HTTP and/or Reflector acces?

PS: see also my other answer here Matt:

viewtopic.php?f=131&t=26965&p=216717#p216717
(last post)

[matt (support)]

It sounds like something is in a bad state. Have you tried a clean restart of the Mac (and Indigo Server of course) to see if that helps?

It sounds like something is in a bad state. Have you tried a clean restart of the Mac (and Indigo Server of course) to see if that helps?

Thanks for the patient support.
Have restarted the Mac mini.
But that doesn't solve the problem:
Both locally and remotely the load of a control page is very slooow.
Please note: before the update to Indigo 2022.2.0 the load of a control page was fast and smooth. So the Mac and network are not the problem.

What stands out is that specially when the page is starting to load in Safari, the blue status bar in the URL window keeps hanging for ~ 8 seconds at ~ 10%.
Nothing happens then. And then out of nothing the status bar goes further to 100% and then the page is loaded but the 5 thumbnails of the live streams of the cams come after it and are popping up one by one in a couple of seconds.
As said before; we never had this with previous versions of Indigo.

Activity monitor says: SecuritySpy 20.4% CPU load (the heaviest) and then a looong list of IndigoPluginHosts3 with 15.3 - 0.1 % and the IndigoServer is between 4 and 10%.

Even a simple and small control page with a couple of items takes a ~ 8 seconds to load.
And the On/Off feedback of a "PlainButton_small" at that control page takes ~ 10 seconds before it becomes green or red.
(It takes ~4 seconds when the lamp goes on or off. The HomeKitLink Siri plugin switches the lamp on within a second when we ask Siri to switch the lamp on).

Sometimes (not always) we saw the errors below into the log.

Hope this story helps to solve the problem, thanks for your effort.

Web Server Warning HTTP 503 error for request /web/js/indigoajax.js from 127.0.0.1
Web Server Warning undefined page for request /web/js/dialog_pa.js from 127.0.0.1
Web Server Warning HTTP 503 error for request /web/js/dialog_pa.js from 127.0.0.1
Web Server Warning undefined page for request /web/js/indigoutil.js from 127.0.0.1
Web Server Warning HTTP 503 error for request /web/js/indigoutil.js from 127.0.0.1
Web Server Warning undefined page for request /images/controls/devices/PlainButton_small+on.png from 127.0.0.1
Web Server Warning HTTP 503 error for request /images/controls/devices/PlainButton_small+on.png from 127.0.0.1
Web Server Warning undefined page for request /images/controls/variables/dot_blue+true.png from 127.0.0.1
Web Server Warning HTTP 503 error for request /images/controls/variables/dot_blue+true.png from 127.0.0.1
Web Server Warning undefined page for request /images/controls/variables/dot+flame.png from 127.0.0.1
Web Server Warning HTTP 503 error for request /images/controls/variables/dot+flame.png from 127.0.0.1

[matt (support)]

After investigating this we believe the problem is caused when there are multiple clients (Safari, Indigo Touch, etc.) requesting updates from the Web server (likely for Control Pages). It might also happen just with one client, given a sufficiently complex Control Page. We think having refreshing image controls on CPs, especially if they are requesting images from external servers or other applications (like SecuritySpy). might cause the Indigo Web server to become far enough behind on incoming requests that performance significantly degrades and eventually those 503 errors start to occur. Note when those 503 errors occur it means an HTTP request to Indigo's Web server has taken over 60 seconds to complete. That may not be a request for the entire CP, but could be for just a single refreshing image inside of a CP. In the case of your Event Log snippet above it is simple requests that are failing for static files/images. For that to be the case I think the entire Web server must be really bogged down (likely on CP updates).

We haven't yet been able to reproduce the problem here, but are actively working on what we think will fix the issue.

Thank you so much for the extended explanation.

We also thought that the problem was caused by multiple clients at the same time, but even 1 client on the local Mac is already causing the problem.
So there are 2 problems:
- The slow building of a (complex) control page.
(the requesting images are not coming from SecuritySpy, these are URL's directly from the cams)
- But also the feedback from a simple control page with 1 item (a lamp).
When it's turned on at that simple control page, it takes more than 3-10 seconds before the image of the lamp is be refreshed.
(the lamp does turn on/off quickly)

With Indigo Touch at an iPhone the feedback of the Control Pages are a little bit faster but still much slower than previous Indigo versions.

Two questions;

- How much time do you need to solve the problem?
- What is the best procedure to downgrade the Indigo Server again?
(because this is unfortunately unworkable in daily practice, a new version should run faster, not slower)

I understand that you have put a lot of time and energy into this update and that this has been a disappointing experience.

Sorry about that.

[matt (support)]

I think if you close all the browsers/clients, restart the Indigo Server, then you'll find that the simple case (feedback of a control page with a lamp on it) is very fast. I think the Web server is getting behind on requests because of the multiple clients / complex CP and then doesn't get back into a good state (without an Indigo Server restart). If you find that isn't the case then let us know so we can try to reproduce it here. As it is now, all of the CPs we've tested are very fast and responsive.

The refreshing images to URLs that are cams is likely the culprit, if I had to guess.

We don't have an ETA on a fix but we are working on it and will package up the release shortly after we have it (a few other things are going into it as well). So it won't be months for sure, but will likely not be tomorrow either.

You can shutdown the Indigo 2022.2 server and quit the client, then just relaunch the Indigo 2021 server (using the 2021 client). It won't back propagate any database or settings changes of course, but reverting to the previous version should be trivial – we intentionally make our feature update versions this way.

I think if you close all the browsers/clients, restart the Indigo Server, then you'll find that the simple case (feedback of a control page with a lamp on it) is very fast..

Thanks for the feedback.
It's indeed faster when we restart the Indigo Server and open only locally this simple page. The feedback is ~ 3 seconds instead of 7-10 seconds.
And it doesn't matter if this is local or remote, feedback time is the same.
Then we did also a test with the control page with 410 items without the 5 cams feeds. That feedback is also ~ 3 seconds, locally and remote.

So you were right that the problem is probably the cam feeds.
But it is still strange that a simple CP with this version of Indigo is slower than the previous version. We have never had this kind of speed problems before, not even with the cam feeds.

Let's hope you will find a solution.

Thanks for all your effort.

[matt (support)]

We tried again and are not able to reproduce this. We've tried several different Control Pages loading on all the major browsers on different Macs, even a 2021 Mac mini 2012 running Sierra, and they all loaded in a fraction of a second.

You mentioned trying local and remote. For troubleshooting lets definitely stick with local connections only. Try it on the same Mac running the Indigo Server (so using localhost:8176) as well as another Mac on the same LAN.

Also try creating a new Indigo database file with a simple Control Page. Perhaps the problem is related to the complexity of your database (even though you tried a simple CP).

If that doesn't work, then try disabling all of your plugins temporarily. Just shutdown the Indigo Server and just move all the plugins to another folder temporarily.

[jay (support)]

The details of our testing: attempted to recreate your 3+ second load time on a simple page (one element) from a slow Mac client (2012 mini running Sierra) to a slowish Indigo Server (2014 mini running Mojave) and the connection goes through the router from WiFi to LAN and back. I've seen no delay over a second, and most are pretty much instantaneous. I've also tested Safari, Chrome, FireFox, Opera from my dev mac to the same Indigo Server and all were sub one second load times.

Thanks for both responses.

The fact remains that with previous Indigo versions there were no problems with speed at all.
The problems have come with this latest 2022.2.0 version. There is definitely something changed.

We are willing to cooperate to find out where the problem is, but this must remain realistic.
We cannot just play trial and error with a well-functioning database and all Python3 plugins that normally functions excellently.

With an M1 Mac mini and Mac Os Monterey the most heaviest application is SecuritySpy (20.4%).
So, the hardware is perfect. The Indigo database is 7.6Mb.
Is this too big for Indigo version 2022.2.0?
(Or is our Database too complex for Indigo 2022.2.0?)

It's a pity that you can't reproduce the problem. So, the most simple solution is going back to the former version.
This was working stable and we can use it for the next coming years.

[matt (support)]

John,

You can continue to bewail about the situation, but the fact of the matter is you are currently the only user that has reported this problem and we are unable to reproduce it. I have no doubt the problem is related to 2022.2 but if we cannot reproduce it and you won't troubleshoot it (and we don't have other reports), then there is nothing else that can be done. Either you will have to troubleshoot it to narrow down the problem or you will need to roll back to Indigo 2022.1, or find a different solution altogether.

And let's be honest here, the steps I asked you to do above can be done in a few minutes. Note Jay and I have spent hours discussing the problem, trying to reproduce it, and investigating code changes that might fix it. Unfortunately, the code changes that we had hoped would fix it won't fix the case of a single simple CP (with no refreshing image controls) being slow. Therefore, without more troubleshooting to narrow down the problem we probably won't implement them as they are non-trivial and will require significant QA work.

So, it took some effort, but we are happy again!
(An Indigo database of 4 days old also had to be updated with the latest energy variables).

The previous version of Indigo works super fast: (lamp is on within milliseconds) and the buildup of the control pages (with 5 cam feeds) is also super fast!

Sorry that we, as loooong-time regular customers, are lamenting according to you.
Personally, as an Apple reseller with 35 years of experience, I would never dare to say that publicly to a regular customer.

We understand that it is also difficult for you to solve such a problem remotely.
That is why we have previously offered to do a personal remote session with you, but you didn't want that.
We also do this a lot with customers and also with a number of plugin developers.
That is often more direct and personal.
We understand the charm of the forum, which is also useful, but sometimes it is wiser to solve technical (difficult) challenges in person first and not in public.

With 7 users and a lot of (energy) variables isn't it easy to shut down an Indigo Server that is used 24 hours a day and then play trial and error. Apart from some plugins that can no longer find the device data and then create all kinds of new data. Which then causes problems with the existing database afterwards.
We would like to be guinea pigs, but then we would have liked to have known that earlier because then we would have built a separate extra Indigo Server.
And we immediately believe that it works perfectly and super fast on your side with the new version. That will be here also with a new database.
But we work with a database of CPs with more than 400 items that are used every day. And that must continue to work super fast and reliably, also with future Indigo versions.

We don't need a response because then we will continue to discuss (publicly).

At the end of the day we are happy again with our existing Indigo version!

Thanks for your effort and time.