Driver disconnects/restarts many times during the day and HA integration failes to connect

[zavjah]

Checklist

• I have read and followed the above instructions

• I have checked the troubleshooting section and my problem is not described there.

• I have read the changelog and my problem was not mentioned there or the fix did not work.

Describe the issue

Hi folks,

I am in desperate need for help :-(.

Something seems to be wrong with my zwave_js as zwave integration fails to connect many times during the day. Sometimes zwave_js recovers and starts working again but often I am left with no other option but to reboot the system to get it work again.

I've reported this in HA core forum and I was directed to look into the addon itself. Did so and every time HA cannot connect to zwave-server strange things are in the zwave_js log. I've tried to dig into the zwave_js log my self and I did found our some strange things. Nevertheless I have no clue what this all means. Here are two things:

1. A snippet of my zwave_js debug log between two restarts, as i understand it (log entry: starting driver...)
2. diagnostics of my zwave integration from HA

This is what I see:

1. zwave_JS was started
2. lots of actions were performed and suddenly soft resets is performed at timestamp 09.52:54, although I have disabled soft reset in the zwave settings.
3. controller reconnected and nodes were pinged and other stuf
4. Then driver executed GetPowerlevel16Bit but the controller reported "Failed to execute controller command after 1/3 attempts. Scheduling next try in 100 ms." (timestamp 09:53:23.405Z)
5. This GetPowerlevel16Bit was executed few times more and when driver received RES it reacted with [unexpected] few times. (timestamp 09:53:58.470Z). End of story was another "Failed to execute..." at 09:54:47.418Z
6. This went on few times more after this and driver discarded the response
7. Then again command GetRFRegion was sent and each response was flagged [unexpected]
8. Again, at 09:55:48.289Z soft reset was performed and
9. at 09:56:13.303Z serial received lots of responses, there was a CAN in the middle and at the end, driver reported "Dropping message because it could not be deserialized..." many times
10. The end of story was that at 9:58:01.690Z driver start destroying the driver instance and driver started again :-(

PLEASE, can someone help me understand what is wrong and what I can do about it.thx

Software versions

Driver (node-zwave-js): 12.4.4
Z-Wave JS UI: 9.9.1
Home Assistant Z-Wave Integration: 2024.4.0
Home Assistant Z-Wave JS UI Addon: 3.4.1

Device information

Manufacturer: Z-Wave.me
Model name: Razberry7
Node ID: 1

Checklist

• I made sure to provide a driver log on level debug.

• The log includes a re-interview of the problematic device (if applicable).

• The log includes the problematic interaction with the device (if applicable).

• I provided the node ID of the problematic device (if applicable).

Upload Logfile

zwave_js-integration_diagnistics_240404_160000.json
zwavejs_current_240404_170711.log

[AlCalzone]

It seems that you're misunderstanding some things:

suddenly soft resets is performed at timestamp 09.52:54, although I have disabled soft reset in the zwave settings

Soft-reset is always enabled for 700/800 series. Disabling does nothing for them. The soft reset during startup is required to get the controller into a defined state.

controller reconnected and nodes were pinged and other stuf

It doesn't look like they were actually pinged yet - or the logs are incomplete. In general I'm missing a few things here and there. There are also quite some delays between the log lines, not sure where they come from. Plus tons of repeated incoming commands. This all looks a bit odd - is your system under heavy load? Maybe the serial communication is not working reliably due to this.

but the controller reported "Failed to execute controller command after 1/3 attempts. Scheduling next try in 100 ms."

The controller didn't respond at all within the expected timeouts, that's why Z-Wave JS logs this message.

---

Anyways, it seems that the unresponsive controller detection in Z-Wave JS triggers a few times during the log, but it doesn't have much success in recovering. I'm not sure if this is because of the known bug the 7.19.x SDK (#6512 and similar issues) that causes the controller to hang, or high system load interfering with the serial communication, or both.

Few things you could try:

• Move to a different system (to check if that's the issue)
• Disable the unresponsive controller detection in the settings (as a band-aid)
• Wait for an updated firmware - the fix is supposed to be released mid April

[zavjah]

Hello @AlCalzone,

thx a lot for your reply and your explanation of soft reset. Got that now.

1. Log attached was just an excerpt from A=>B but in between I have not deleted any lines. Here is the entire zwave_js_current.log:
   zwave-js-ui-store.zip

   How does this one look like?

2. You asked if my system is under heavy load? Tbh, I do not know! How can I determine this? by looking into the statistics of the controller or every node?

3. What do you mean by "Move to a different system (to check if that's the issue)"?
   Away from ZwaveJS UI and Home Assistant or something else?

4. Is it this setting you are referring to with "Disable the unresponsive controller detection in the settings (as a band-aid)":
   

5. The firmware update you are talking about is imho for an Aeotec Z-Stick 7 Plus, which I do not use. I have a ReZberry 7 directly attached to my Pi3 board, which has FW 7.36 / 07.19.02.01 SDK 7.19.02. There is though an update to 7.38 / 07.20.00.00 SDK 7.20.00 that fixes network promisc mode always enabled and updated to the new SiLabs SDK.
   Is my understanding correct here? If so how does this fits into me having this controller since 02.23 and problems started first around 09.23?

Please excuse if I am posting silly questions but I am new to this but a fast learner :-D
thx a lot

[zavjah]

Hello @AlCalzone,

your last comment explains a lot and finally sheds some light into the shadows I wasn't able to clear since my problems arose. THANK YOU. I've disabled the option you suggested, let's see, if this helps. I cannot remember having dead nodes before that so I hope it'll help

Also, thx for the clarification about the FW. I am going to install ZWAVE.me on a different SD-card and boot it on the same PI so I can do OTA update of FW once it is out. Unfortunately, ASFAIK. they do not offer the FW as a file download to use ZWave JS UI for Update so I must do it this way.

In regards to heavy load:

• CPU usage is just about 20-30%, so that shouldn't be the issue.
• But, as I have a pi3b+ with 1GB RAM which of app. 80-90% are all the time spent, I am not sure, if that is OK.
• I have 58 ZWave nodes in my network, most of them switches power supplied, and a Raspberry7 module. But also four reolink cameras, a weather station and few ZigBee devices, just to give you an impression.

I am going to upgrade to Pi4 with 4GB to exclude to small Pi playing a role in this.

Furthermore, I want also to look into the statistics to see if there are nodes sending more messages than the other. But I do not know how to interpret those metrics here:

Is there a documentation I can read into this or can you tell which of those could indicate a "rogue" node.

thx a lot and cheers,
zavjah

[AlCalzone]

I do not know how to interpret those metrics here:

It seems like you're looking at the controller statistics. Each device has its own statistics.

[zavjah]

Yes, I was looking at the controller and I know each node has the same. But what do they mean?

[kpine]

But, as I have a pi3b+ with 1GB RAM which of app. 80-90% are all the time spent, I am not sure, if that is OK.

80-90% memory usage is too high IMO. Z-Wave JS UI itself requires ~100-200MB on average (in my experience), that's already ~10% of your total RAM. You are at risk of Z-Wave JS UI being OOM killed (out of memory). I didn't see it in any of your logs, but it's still a possibility. I don't know if HAOS allows you to see the kernel logs, but that could show something. Other users have had problems with this much memory used.

[zavjah]

Hello @kpine, thx for you answer. I've already ordered pi4 with 4GB RAM. Will switch the hw next week Nd see where I'm getting.