I’ve had this happen when I forgot that a device was part of a trigger. You could try right clicking the device and showing the dependencies and see if that’s your issue.
That was the problem. I had the fan defined as a member of an action group. Once I deleted it from the action group it worked correctly. (My mistake was moving the z-wave switch from one device to another that should not have been a member of the group.)
I replied too soon. It still is not working consistently. It appears to be a problem with the fan because I switched the fan to a different device and it still doesn't stay off.