Indigo Facial Recognition via Node-Red

Ideally, I'd just use the python facial recognition stuff, but it won't let me tinker without using a 3rd party Python 3.XXX which is beyond my skill level. So, I figured out how to do it in my node red server with runs on my indigo machine (to handle a few odds and ends)

The Nodes I used for this:

node-red-contrib-indigo. https://flows.nodered.org/node/node-red-contrib-indigo
node-red-contrib-facial-recognition https://flows.nodered.org/node/node-red-contrib-facial-recognition
node-red-contrib-image-output https://flows.nodered.org/node/node-red-contrib-image-output
node-red-node-annotate-image https://flows.nodered.org/node/node-red-node-annotate-image

On the backend, you'll need:

1 action to start the node-red flow.
6 variables for flow data to go into
1 "KnownFaces" folder on your desktop. The subfolders are labeled "Firstname Lastname" with pictures of those individuals faces inside the corresponding folder.
1 "UnknownFaces" folder on your desktop
Some other mechanism of your choosing to capture an image and save as "UKF.jpg" into the "UnknownFaces" folder

The process:

IP Camera script saves a snapshot and puts a copy "UKF.jpg" into my "UnknownFaces" folder.
An action group updates the HTTP In node in my Facial Recognition Flow. The action just executes this script:

Code: Select all

import requests
r = requests.get('http://192.168.1.10:1880/FaceRecognizer?recognize=all')
r.raise_for_status()

Running the action starts the Node-Red flow.
The facial recognizer node does its magic and then does a couple things.

1) annotates the image with a box around each face and a name of all recognized faces.
2) saves a copy of the annotated image into the "UnknownFaces" folder with the name "Found.jpg" <-- overwritten whenever this is run.
3) splits out the message for the first two people recognized and updates Indigo Variables with that information.

Screen Shot 2021-01-13 at 6.49.45 PM.png (28.51 KiB) Viewed 2219 times

Screen Shot 2021-01-13 at 6.53.29 PM.png (218.11 KiB) Viewed 2219 times

If you import this into your node-red, that should give you a leg up on the process if you are going to set it up the way I did:

Code: Select all

[{"id":"83d559af.a0ed6","type":"tab","label":"Facial Recognition","disabled":false,"info":""},{"id":"dd4c55d0.245","type":"http in","z":"83d559af.a0ed6","name":"192.168.1.10:1880/FaceRecognizer","url":"/FaceRecognizer","method":"get","upload":false,"swaggerDoc":"","x":160,"y":240,"wires":[["9c39ee04.5a9a7","d640e2e5.927328"]]},{"id":"9c39ee04.5a9a7","type":"http response","z":"83d559af.a0ed6","name":"","statusCode":"","headers":{},"x":390,"y":300,"wires":[]},{"id":"d640e2e5.927328","type":"json","z":"83d559af.a0ed6","name":"","property":"payload","action":"","pretty":false,"x":390,"y":260,"wires":[["3fce7ac2.2026be"]]},{"id":"3fce7ac2.2026be","type":"switch","z":"83d559af.a0ed6","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"all","vt":"str"},{"t":"cont","v":"off","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":510,"y":260,"wires":[["a1fcdc7f.dafe8"],["376864a.dc6831c"]]},{"id":"a1fcdc7f.dafe8","type":"change","z":"83d559af.a0ed6","name":"ON","rules":[{"t":"set","p":"payload","pt":"msg","to":"ON","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":650,"y":220,"wires":[["f363ff4e.11ea58"]]},{"id":"376864a.dc6831c","type":"change","z":"83d559af.a0ed6","name":"OFF","rules":[{"t":"set","p":"payload","pt":"msg","to":"OFF","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":650,"y":300,"wires":[[]]},{"id":"f363ff4e.11ea58","type":"file in","z":"83d559af.a0ed6","name":"","filename":"/users/yourname/Desktop/UnknownFaces/UKF.jpg","format":"","chunk":false,"sendError":false,"encoding":"none","x":300,"y":460,"wires":[["da7daaf8.264a38"]]},{"id":"d394509a.939b3","type":"function","z":"83d559af.a0ed6","name":"Prep rects for annotate image node","func":"//global vars\nvar the_rects;\n\n//was detectAllFaces or detectSingleFace used\n//check to see if payload.Result is an array (detectAllFaces)\nvar Result = msg.payload.Result;\nif ( Array.isArray(Result) ) {\n    // get just the rect values and place in array\n    the_rects = Result.map(x => {\n        //check for label from FaceRecognition\n        var match_label;\n        if ( x.match ) {\n            match_label = x.match._label;\n        }\n        else {\n            match_label = \"\";\n        }\n        var result = {\n            type: \"rect\",\n            x: x.detection._box._x,\n            y: x.detection._box._y,\n            w: x.detection._box._width,\n            h: x.detection._box._height,\n            label: match_label\n        }\n        return result;\n    });\n    msg.annotations = the_rects;\n}\n//else detectSingleFace was used\nelse {\n    //check for label from FaceRecognition\n        var match_label;\n        if ( Result.match ) {\n            match_label = Result.match._label;\n        }\n        else {\n            match_label = \"\";\n        }\n    the_rects = [{\n        type: \"rect\",\n        x: Result.detection._box._x,\n        y: Result.detection._box._y,\n        w: Result.detection._box._width,\n        h: Result.detection._box._height,\n        label: match_label.match._label\n    }]\n    msg.annotations = the_rects;\n}\n\n\n\n//var xx = msg.payload.Result[0].detection._box._x;\n//var yy = msg.payload.Result[0].detection._box._y;\n//var ww = msg.payload.Result[0].detection._box._width;\n//var hh = msg.payload.Result[0].detection._box._height;\n\n//msg.annotations = [ {\n//    type: \"rect\",\n//    //bbox: [ xx, yy, ww, hh],\n//    x: xx, y: yy, w: ww, h: hh,\n//    //bbox: [ 100, 100, 100, 100],\n//    label: \"Tara Sanford\"\n//}]\n\nmsg.payload = msg.payload.OriginalBufferedImg;\n\nreturn msg;\n\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":980,"y":340,"wires":[["a704c0e3.d1086"]]},{"id":"a704c0e3.d1086","type":"annotate-image","z":"83d559af.a0ed6","name":"","fill":"","stroke":"#0070c0","lineWidth":"20","fontSize":"48","fontColor":"#0070c0","x":900,"y":420,"wires":[["7c0b0a99.fba74c","d4d0bc7f.43516"]]},{"id":"da7daaf8.264a38","type":"facial-recognition","z":"83d559af.a0ed6","image":"payload","settings":"settings","name":"","bindings":"CPU","FaceDetector":"SsdMobilenetv1","FaceDetector_SsdMobilenetv1_maxResults":"5","FaceDetector_SsdMobilenetv1_minConfidence":"0.6","FaceDetector_tinyFaceDetector_inputSize":"416","FaceDetector_tinyFaceDetector_scoreThreshold":"0.5","Tasks":"detectAllFaces","FaceLandmarks":true,"FaceExpressions":true,"AgeAndGender":true,"FaceDescriptors":true,"Face_Recognition":"Face_Recognition_enabled","Face_Recognition_enabled_path":"/users/williammoore/Desktop/KnownFaces","Face_Recognition_distanceThreshold":0.7,"x":630,"y":460,"wires":[["d394509a.939b3","fa677f40.aa8388","72039a5d.15b924","fcbfdb46.c7f768","f6133bfe.14f7d","28c8f510.8dd42a","5018bb5.11054c4"]]},{"id":"7c0b0a99.fba74c","type":"file","z":"83d559af.a0ed6","name":"Save Processed Image","filename":"/users/yourname/Desktop/UnknownFaces/Found.jpg","appendNewline":true,"createDir":false,"overwriteFile":"true","encoding":"none","x":940,"y":480,"wires":[[]]},{"id":"541278cb.d7866","type":"inject","z":"83d559af.a0ed6","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":100,"y":380,"wires":[["f363ff4e.11ea58"]]},{"id":"82c1232.b9d616","type":"comment","z":"83d559af.a0ed6","name":"Launched From Indigo Action","info":"","x":140,"y":200,"wires":[]},{"id":"d4d0bc7f.43516","type":"image","z":"83d559af.a0ed6","name":"","width":"800","data":"payload","dataType":"msg","thumbnail":false,"active":true,"pass":true,"outputs":1,"x":840,"y":540,"wires":[[]]},{"id":"2cf91b2f.11b27c","type":"indigo-set-variable","z":"83d559af.a0ed6","name":"Person 0 Name","controller":"2387b996.34c33e","itemname":"Person0MatchLable","x":480,"y":580,"wires":[[]]},{"id":"fa677f40.aa8388","type":"change","z":"83d559af.a0ed6","name":"Person 0 Name","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.Result[0].match._label","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":280,"y":580,"wires":[["2cf91b2f.11b27c"]]},{"id":"c7279453.56e3f","type":"indigo-set-variable","z":"83d559af.a0ed6","name":"Person 0 Gender","controller":"2387b996.34c33e","itemname":"Person0Gender","x":490,"y":620,"wires":[[]]},{"id":"602c2ef5.0e903","type":"indigo-set-variable","z":"83d559af.a0ed6","name":"Person 0 Age","controller":"2387b996.34c33e","itemname":"Person0Age","x":470,"y":660,"wires":[[]]},{"id":"72039a5d.15b924","type":"change","z":"83d559af.a0ed6","name":"Person 0 Gender","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.Result[0].gender","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":620,"wires":[["c7279453.56e3f"]]},{"id":"fcbfdb46.c7f768","type":"change","z":"83d559af.a0ed6","name":"Person 0 Age","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.Result[0].age","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":270,"y":660,"wires":[["602c2ef5.0e903"]]},{"id":"dff3f647.fdb778","type":"indigo-set-variable","z":"83d559af.a0ed6","name":"Person 1 Name","controller":"2387b996.34c33e","itemname":"Person1MatchLable","x":480,"y":740,"wires":[[]]},{"id":"f6133bfe.14f7d","type":"change","z":"83d559af.a0ed6","name":"Person 1 Name","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.Result[1].match._label","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":280,"y":740,"wires":[["dff3f647.fdb778"]]},{"id":"3befb188.164216","type":"indigo-set-variable","z":"83d559af.a0ed6","name":"Person 1 Gender","controller":"2387b996.34c33e","itemname":"Person1Gender","x":490,"y":780,"wires":[[]]},{"id":"2df1b09.f9a1d5","type":"indigo-set-variable","z":"83d559af.a0ed6","name":"Person 1 Age","controller":"2387b996.34c33e","itemname":"Person1Age","x":470,"y":820,"wires":[[]]},{"id":"28c8f510.8dd42a","type":"change","z":"83d559af.a0ed6","name":"Person 1 Gender","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.Result[1].gender","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":780,"wires":[["3befb188.164216"]]},{"id":"5018bb5.11054c4","type":"change","z":"83d559af.a0ed6","name":"Person 1 Age","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.Result[1].age","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":270,"y":820,"wires":[["2df1b09.f9a1d5"]]},{"id":"d7d6e5fd.570ab8","type":"comment","z":"83d559af.a0ed6","name":"Use Timestamp to test your work","info":"","x":170,"y":340,"wires":[]},{"id":"bd48f09a.94a22","type":"comment","z":"83d559af.a0ed6","name":"Set location folder of Known Images and file path for unknown","info":"","x":580,"y":500,"wires":[]},{"id":"554249d3.e22a3","type":"comment","z":"83d559af.a0ed6","name":"Image preview isn't needed, good for testing","info":"","x":1090,"y":380,"wires":[]},{"id":"2387b996.34c33e","type":"indigo-controller","name":"INDIGOSERVER","protocol":"http","host":"HOSTNAME.local.","port":"8176","username":"","password":""}]

Happy Tinkering!!

Nice work!

Nice work!

Thank you. I think you did a lot of the work with the Alexa integration java stuff?.... There is a .js version of this facial recognition stuff that I think could be used instead of Node-Red as a direct implementation... but I don't know java.

https://github.com/justadudewhohacks/face-api.js/

And all the python versions I found required libraries things that required python 3.x (and I don't want to risk breaking something I don't know how to fix)

Holy cow man, looks like this may be the sign that I should finally pick up node red. Thanks for pulling me back in!

And this is what it looks like now....

Screen Shot 2021-01-19 at 6.06.24 PM.png (245.05 KiB) Viewed 2075 times

Obviously, it got a little crazy, but I have it working with either a snapshot image placed in a desktop folder or as a constant (every 1 second) pull from a camera image directly into the flow. It sets variables in indigo for "InputHasFace", "InputHasKnown" and "InputHasUnknown". <-- Populating with "False" is okay if doing a single snapshot image, but if your watching a camera, you will have a ton of "False" variable updates taking over your log. I also added a "Gate". So the Inject node is always on, but there is a "Gate" between that and the HTTP-In node. I have indigo actions that open or close the "Gate" and if a known face is found, the gate is closed.

Case use -> camera or PIR detect motion. Trigger opens the gate for that camera for 5 minutes. Once a known face is recognized, the gate closes. Once 5 minutes passes, gate closes.

I also separated it so it creates a couple different images with the boxes and labels on them. For Known, it replaces a "Found_Face.jpg" and creates a new "Found_Face_Date-Time.jpg" in a log folder. If a face is "Unknown", it replaces a "Unknown_Face.jpg" and creates a new "Unknown_Face_Date-Time.jpg" in a different log.

I don't trust it enough to use for direct access control, but maybe as a 2-factor thing.... right now it's just something to mess with the kids.

Probably my last test of this before moving to something else.... and what it looks like today:

Screen Shot 2021-01-21 at 4.04.43 PM.png (164.16 KiB) Viewed 2036 times

With the previous, there was WAY too much indigo traffic. With every capture or attempt to capture there was logs of variables changing from true to false or false to false, etc. So I moved it all around. It's hard to see, but I'll try to explain. Far far left is a dahua node that talks to my NVR and receives motion alerts. Those alerts open/close the gates that request snapshots to be sent to the facial recognition engine. I have them all set up to create saved image files, only a couple of the cameras are used to update variables.

Here is the issue... the facial recognition is entertaining, but not a reliable method for anything you would automate (in my opinion). Even with the confidence dialed up to 0.8, it though I looked like Sandra Oh (Christina Yang on Grey's Anatomy)

FoundMatch_1611172247013rz.jpg (114.23 KiB) Viewed 2036 times

The found face works pretty well as long as the person is still enough or the image isn't blurred, with the occasional TV image, picture in the background or the occasional HomeDepot Bucket?

Bucket.png (15.47 KiB) Viewed 2036 times