Unit Testing

Posted on
Wed Feb 24, 2021 9:10 am
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Unit Testing

I thought a thread dedicated to unit testing plugins might be valuable (and unabashedly self-serving). It doesn't seem that it's a big topic of discussion, which surprises me, and I only found a couple threads that mention it. Personally, I am only starting to scratch the surface with this stuff, and am cutting my teeth on a non-Indigo side project. But I can clearly see a value to implementing it with my plugins and--as Jay's post mentions--PyCharm has nice support for the concept so integrating it into my workflow should be pretty easy.

viewtopic.php?f=330&t=23697&p=192084&hilit=unittest#p192084
viewtopic.php?f=107&t=15676&p=113127&hilit=unittest#p113127 [unfortunately, the Github link on this post is no longer valid]

If anyone has any plugin unit testing tips--especially with respect to Mocking and/or Monkey patching--they would be greatly appreciated.

EDIT: found user gazally's referenced test code here: https://github.com/gazally/indigo-omnil ... lugin/test

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Wed Feb 24, 2021 3:56 pm
RogueProeliator offline
User avatar
Posts: 2501
Joined: Nov 13, 2012
Location: Baton Rouge, LA

Re: Unit Testing

Oooh, good topic, Dave!

I've recently been working on a bunch of new code for Domotics Pad and other projects that involves re-writing the Indigo network interfaces; though this is not the Python for the plugins, I can share a little that might be relevant. First off, you might make a distinction between unit testing and integration testing -- both are important and both have their merits (and there are various semantics here, conventions, etc. that some people do or will argue over, but this is my practical approach to it).

Unit Testing
Test everything that you can possibly and reasonably test... I am not a TDD fiend like some and don't necessarily try for 100% coverage of every single line out there, but there is merit to having coverage over as much as is practical and, importantly, that you want to be able to regression test upon making a change. These tests should be quick (so they can be run before every push to your repo, if not every commit) and everything stay local. Make use of mocked objects/interfaces/etc.

Integration Testing
Honestly, this is a pretty new addition to my code, especially in the Domotics Pad/Indigo world, but it definitely gives me a sense of confidence that things will work "in the real world". My integration tests take longer to run, do actual network and device access, and require a little more setup. If I am going to run these (and this can be automated of course), I make sure that my Dev Indigo Server is in a valid state (in terms of the initial setup) by executing an Action Group which initializes everything to the proper state to begin. Then I allow the tests to run against the dev box; some can self verify in the test itself, so I verify on the Indigo box after the tests complete.

There are many ways to do this, but one easy way is to have two unit test projects - one is for unit tests and one are the unit tests that are acting as integration tests. I could see a similar approach working for plugins -- even potentially having integration tests that hit real devices when practical.

I am curious what others have to say on this topic as I have never done any Python automated testing.... grabbing popcorn and waiting! :-)

Posted on
Wed Feb 24, 2021 5:36 pm
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: Unit Testing

Thanks for contributing, brother. I'm at a place where I will definitely reap more than I sow--at least for a while--but have started to come to grips.

My approach to this at the outset will be to use the built-in unittest library. What I've read so far says that it's easier for a beginner to pick up and (apparently) unittests are compatible with pytest--so there should be no trouble switching later if needed. I never envisioned doing every line of code, although I could see that being useful in a commercial project. It will take some time to roll out across my plugins, and I though I'd start with low hanging fruit like typing, math, I/O, and similar obvious stuff. Then, when errors crop up in development, I plan to add a test to cover that circumstance and build a library of tests over time. For example, I shared a build of a side project with Jon (autolog) and when he ran it, the thing wouldn't even load--because of one missing comma in a json file. I'm now testing for this. :D

With Integration testing, I'm thinking things like programmatically creating Indigo objects and running them through their paces in an automated way. What I do now is create devices with intentional challenges to see how they react; but I have to run these tests manually. Somewhere in here (or maybe in between) is tests that change the code for the test and then change it back. One example is the Matplotlib plugin where I have to simulate battery operated devices because I have no devices physically connected to my development environment. I don't do this by simulating an Indigo device object; rather, I just pass a phony dict of what Indigo *would* provide if one existed. Again, I do this manually (and often forget to change the code back when I'm done).

Over time, I can see this saving a lot of time and effort.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Sat Feb 27, 2021 12:53 pm
aaronlionsheep offline
Posts: 260
Joined: Feb 24, 2019
Location: Virginia, USA

Re: Unit Testing

Wow I wish I had found gazally's repo before I hacked together my own mocking library of Indigo.

I personally love TDD, although this is because it was drilled into me during school (submitted projects required 100% code coverage by unit tests). When I built ShellyMQTT, I didn't have access to all of the devices. I found that writing unit tests let me be confident that all of the small components worked and that I could simply integrate these components together to define the classes that represent the Shelly models.

Pedantically speaking, a unit test should comprise of a single assert (basically test ONE thing). I personally try and limit the scope of my unit tests, but I blur the lines between a unit test and integration test. For example, some of my unit tests include a full message of what a Shelly device might send to my plugin and I assert all the states the device should be in after parsing that message. I use mocking to force-feed my message into my device class so that I can ensure my device class sets the proper states and all that fun stuff.

With the amount of class inheritance in my plugin, unit tests have save my butt a few times. I made a change to an inherited method and my unit tests caught that a device no longer worked as it had before. So that was nice to see that my work writing those test had finally paid off.

It would be really nice if unit testing was part of a library offered by indigo, but I may look at using gazally's mocking tools that were created. I used the built-in UnitTest library, although I made extensive use of the UnitTest.mock (?) library to mock all of the Indigo behaviors. I'd like to switch to pytest, but I don't know if I have the strength to rewrite my thousands of lines of test code.

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 6 guests