Unable to remove Zombie Node stuck interviewing.

[jduv]

Checklist

• I have read and followed the above instructions

• I have checked the troubleshooting section and my problem is not described there.

• I have read the changelog and my problem was not mentioned there or the fix did not work.

Describe the issue

What is happening?

I think I've figured out how to put the system into a wonky state. I have a "zombie" node that is stuck in "ProtocolInfo" interviewing state forever and I have no way of getting it out of that state. This includes hard resetting the device (not the stick) and attempting a start smart provision.

What did you expect to happen instead?

If a node is unreachable then I'd expect you to not be able to interview the node, hence putting the node sideways permanently. I'd also expect that I should be able to hard remove a device in some way when it's in this bad state. The device cannot be pinged, but during the "remove node" step the system declares that the node isn't dead.

When I ping this node I get:

2024-04-09 13:48:01.128 INFO Z-WAVE: Calling api pingNode with args: [ 16, [length]: 1 ]
2024-04-09T17:48:01.131Z CNTRLR » [Node 016] pinging the node...
2024-04-09T17:48:01.138Z DRIVER   all queues busy
2024-04-09T17:48:01.143Z SERIAL » 0x010f00a90001001001002500000000167a                                (17 bytes)
2024-04-09T17:48:01.145Z DRIVER » [Node 016] [REQ] [SendDataBridge]
                                  │ source node id:   1
                                  │ transmit options: 0x25
                                  │ callback id:      22
                                  └─[NoOperationCC]
2024-04-09T17:48:01.151Z SERIAL « [ACK]                                                                   (0x06)
2024-04-09T17:48:01.153Z SERIAL « 0x010401a90152                                                       (6 bytes)
2024-04-09T17:48:01.154Z SERIAL » [ACK]                                                                   (0x06)
2024-04-09T17:48:01.157Z DRIVER « [RES] [SendDataBridge]
                                    was sent: true
2024-04-09 13:48:07.170 INFO APP: GET /health/zwave 301 1.315 ms - 191
2024-04-09T17:48:21.160Z SERIAL » 0x01030016ea                                                         (5 bytes)
2024-04-09T17:48:21.164Z DRIVER » [REQ] [SendDataAbort]
2024-04-09T17:48:21.168Z SERIAL « [ACK]                                                                   (0x06)
2024-04-09T17:48:31.161Z CNTRLR   [Node 016] The node is causing the controller to become unresponsive, it is pr
                                  esumed dead
2024-04-09T17:48:31.166Z CNTRLR   [Node 016] ping failed: Timeout while waiting for a callback from the controll
                                  er (ZW0200)
2024-04-09 13:48:31.167 INFO Z-WAVE: Success zwave api call pingNode false
2024-04-09T17:48:31.171Z DRIVER   all queues idle
2024-04-09T17:48:36.175Z DRIVER   all queues busy
2024-04-09T17:48:36.181Z SERIAL » 0x0103003bc7                                                         (5 bytes)
2024-04-09T17:48:36.183Z DRIVER » [REQ] [GetBackgroundRSSI]
2024-04-09T17:48:36.189Z SERIAL « [ACK]                                                                   (0x06)
2024-04-09T17:48:36.191Z SERIAL « 0x0107013b80808080c2                                                 (9 bytes)
2024-04-09T17:48:36.194Z SERIAL » [ACK]                                                                   (0x06)
2024-04-09T17:48:36.196Z DRIVER « [RES] [GetBackgroundRSSI]
                                    channel 0: -128 dBm
                                    channel 1: -128 dBm
                                    channel 2: -128 dBm
                                    channel 3: -128 dBm
2024-04-09T17:48:36.201Z DRIVER   all queues idle
2024-04-09 13:48:37.329 INFO APP: GET /health/zwave 301 1.081 ms - 191

Settings:

Steps to reproduce the behavior:

1. Open up a reasonable zWave installation.
2. Go to a random node in the network and hard reset it.
3. In ZWaveJS UI attempt to interview the node after hard resetting.
4. Observe the node being forever locked in a state you can't recover from.
5. Try to "remove node" and see above log file for the error.

Things I've tried to fix this:

1. Re-hard-resetting the node and provisioning it via start smart:

That didn't work because [node 16] is already zombified.

Is there anything I can do to avoid having to hard reset my stick and rebuild my entire network again? Or is there a way to import a known good configuration after a hard reset? I can just manually remove node 16 in the export I'm guessing.

remove-node.log

Software versions

zwave-js-ui: 9.10.2
zwave-js: 12.5.2

I am using HA's zwaveJS UI add on, latest version.

Device information

Manufacturer: ...
Model name: ...
Node ID: ...

Checklist

• I made sure to provide a driver log on level debug.

• The log includes a re-interview of the problematic device (if applicable).

• The log includes the problematic interaction with the device (if applicable).

• I provided the node ID of the problematic device (if applicable).

Upload Logfile

No response

[zwave-js-bot]

👋 Hey @jduv!

Thanks for opening an issue! It doesn't look like you provided a logfile though. While not strictly necessary for every issue, having a driver log on loglevel debug is required to diagnose most issues.

Please consider uploading a logfile that captures your problem. As a reminder, here's how to create one:

• Z-Wave JS UI
• Home Assistant Z-Wave Integration
• ioBroker.zwave2 Adapter

[jduv]

Log file was attached farther up in the post.

[AlCalzone]

The problem is that removing a failed node requires it to be in the controller's list of failed nodes. This is why we ping first - a failed ping will cause the node to end up there.
However your controller doesn't even start the communication with the node and immediately returns a failure. This causes the node not to end up in the failed nodes list, and the remove failed node command then fails because of it.

Which controller do you have? And which firmware/SDK version?
If it's using a supported SDK version, we can edit the NVM (controller memory) to remove the node.