Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zwave-js-ui crashing after a few days online #4098

Open
3 tasks done
kpanchen opened this issue Jan 20, 2025 · 30 comments
Open
3 tasks done

zwave-js-ui crashing after a few days online #4098

kpanchen opened this issue Jan 20, 2025 · 30 comments
Labels
bug Something isn't working

Comments

@kpanchen
Copy link

kpanchen commented Jan 20, 2025 •
edited
Loading

Checklist

  • I am not using Home Assistant. Or: a developer has told me to come here.
  • I have checked the troubleshooting section and my problem is not described there.
  • I have read the changelog and my problem is not mentioned there.

Deploy method

PKG executable

Z-Wave JS UI version

9.29.1

ZwaveJS version

14.3.7

Describe the bug

Any version after 9.27.7 including 9.29.1 starts crashing after a few days of working normally. Attempted to update to versions: 9.28.0, 9.29.0, both versions starts crashing after running normally for couple days, returned to version 9.27.7 - worked fine for 4 weeks, no issues. Update to version 9.29.1 - crashed again after 2 days running normally.

Logs_20250119.zip

To Reproduce

No particular patterns noted, just updated version crashes after 2-3 days of running normally.

Expected behavior

New versions work.

Additional context

Had a quick look at the log, unfortunately I'm not sure what to check there. However from attached logs: 2025-01-18 - everything worked fine and log ended at the end of day and was archived. I'm not sure at what time 2025-01-19 starts crashing, for whatever reason zwave-ui log is missing and starts at 2025-01-19 22:47:14.701. From that log it is evident that UI was restarted multiple times without any particular reason, it looked like a refresh on the console.

Last crash event happened at 2025-01-19 23:12:58.661 and UI was stopped manually.
2025-01-19 23:17:49.213 - rollback to version 9.27.7 which works fine after. (Marked by keyword "rollback" in the attached logs)

Note: it was a change some time ago where required pointer to configuration (or something like this was missing) and UI was not starting at all, not sure if this is the same issue or something new.

20250119_rollback.zip

No response
@kpanchen kpanchen added the bug Something isn't working label Jan 20, 2025
@codemannh
Copy link

I am experiencing the same issue. Currently running
zwave-js-ui: 9.29.0.b8373a3
zwave-js: 14.3.7
This started when running a previous version which unfortunately I did not capture. When I experienced it, I updated to the version I'm running here after not seeing any issues that looked the same.
A couple of things I have noticed. I have a firmware update for a ZWave extender I have on my network. When I try to update that using the OTA update, it gets to about 79+% done and then ZUI hangs. The only restart is to restart the docker container.
Also, I have a couple of light switch nodes outside my house, still in range and occasionally they go unavailable after switching them on or off from HomeAssistant. My ZWave2JS UI does not run in HA, I run it in a separate docker container.
Don't see anything in any logs I'm collecting. Even upped the logging in ZWaveJSUI to see if it captured any additional info. It didn't seem to.

@robertsLando
Copy link
Member

robertsLando commented Jan 25, 2025 •
edited
Loading

@kpanchen thanks for your issue, did you also tried with 9.27.8? Detecting the exact update that broke this is the first step to fix the issue

@robertsLando
Copy link
Member

@codemannh are you using pkg as well or other deploy method?

@codemannh
Copy link

codemannh commented Jan 25, 2025 via email

@kpanchen
Copy link
Author

kpanchen commented Jan 26, 2025 •
edited
Loading

@kpanchen thanks for your issue, did you also tried with 9.27.8? Detecting the exact update that broke this is the first step to fix the issue

No, will try today. Note that it takes a few days when it starts crashing so I report when I have a result. 9.27.7 works perfectly since I reported the issue and went back to it.

Thank you!!

@kpanchen
Copy link
Author

@kpanchen thanks for your issue, did you also tried with 9.27.8? Detecting the exact update that broke this is the first step to fix the issue

And I was a bit wrong, 9.27.8 crashed pretty much right away. Went back to 9.27.7. Logs attached.

20250125.zip

@Matthew-MBG
Copy link

I wanted to chime in here to say we have been experiencing the same issue, it seems, since sometime in December, although to be fair our installation is on Home Assistant. Regrettably, we don’t know if that started after a Z-Wave JS UI update, but we do know that we updated the Home Assistant add-on on 10 December 2024, and I believe we updated on that day from 3.17 to 3.18. These are the version numbers of the HA add-on, and I am not certain what those version numbers would correspond to in the case of Z-Wave JS UI.

Since then we have been facing Z-Wave JS UI restarts – which we can ascertain are happening because we see them in the log, like so:


[11:21:41] INFO: [32mService Z-Wave JS UI exited with code 256 (by signal 9)[0m

[11:21:42] INFO: [32mStarting the Z-Wave JS UI...[0m

However, we know when it is happening, and have pinned it down to 2 causes in our case:

a) Whenever a device dies. This happens in our network to some devices on the physical periphery of the network. We then have an HA automation which pings them and brings them back to life almost instantaneously, and this has worked decently well for a couple of years. That automation also sends us a notification on our mobile apps. However, we have noticed that since early December, whenever we get one of those notifications, the entire Z-Wave network goes down after a while, and then from the log we can see that Z-Wave JS UI restarted. An HA restart is usually the only way to recover from this, as sometimes it keeps restarting, and even if it starts successfully, the integration with HA does not work until an HA restart.

b) Whenever we open the Z-Wave JS UI interface from within the HA mobile app – there is a good chance that even doing this simple thing causes Z-Wave JS UI to restart – and therefore the same issues as point 1 above ensue. This mostly happens from the app – from the browser it seems to work better (just a feeling – cannot be too sure). In the log, when this is happening, I can see a lot of connection attempts that seem to fail, but I am not qualified enough to understand what that means:


[90m2025-01-25 10:31:12.013[39m [34mDEBUG[39m [1mSOCKET[22m: New connection 3O3-6QwricsA78ycAABR

[90m2025-01-25 10:31:12.014[39m [32mINFO[39m [1mZ-WAVE[22m: Setting user callbacks

[90m2025-01-25 10:31:12.055[39m [34mDEBUG[39m [1mSOCKET[22m: Event INITED emitted to 3O3-6QwricsA78ycAABR

[90m2025-01-25 10:31:12.266[39m [34mDEBUG[39m [1mSOCKET[22m: User disconnected from 3O3-6QwricsA78ycAABR: transport close

[90m2025-01-25 10:31:12.267[39m [32mINFO[39m [1mZ-WAVE[22m: Removing user callbacks

[90m2025-01-25 10:31:13.013[39m [34mDEBUG[39m [1mSOCKET[22m: New connection Hd98_go11_4K_03fAABT

[90m2025-01-25 10:31:13.016[39m [32mINFO[39m [1mZ-WAVE[22m: Setting user callbacks

[90m2025-01-25 10:31:13.191[39m [34mDEBUG[39m [1mSOCKET[22m: Event INITED emitted to Hd98_go11_4K_03fAABT

[90m2025-01-25 10:31:13.305[39m [32mINFO[39m [1mMQTT[22m: MQTT client reconnecting

[90m2025-01-25 10:31:13.462[39m [34mDEBUG[39m [1mSOCKET[22m: User disconnected from Hd98_go11_4K_03fAABT: transport close

[90m2025-01-25 10:31:13.466[39m [32mINFO[39m [1mZ-WAVE[22m: Removing user callbacks

[90m2025-01-25 10:31:14.511[39m [34mDEBUG[39m [1mSOCKET[22m: New connection 5OPMSL_ztSgJQkHUAABV

[90m2025-01-25 10:31:14.514[39m [32mINFO[39m [1mZ-WAVE[22m: Setting user callbacks

[90m2025-01-25 10:31:14.616[39m [34mDEBUG[39m [1mSOCKET[22m: Event INITED emitted to 5OPMSL_ztSgJQkHUAABV

[90m2025-01-25 10:31:15.103[39m [34mDEBUG[39m [1mSOCKET[22m: User disconnected from 5OPMSL_ztSgJQkHUAABV: transport close

[90m2025-01-25 10:31:15.105[39m [32mINFO[39m [1mZ-WAVE[22m: Removing user callbacks

[90m2025-01-25 10:31:16.311[39m [34mDEBUG[39m [1mSOCKET[22m: New connection 8USlO1dysfMGKQoXAABX

[90m2025-01-25 10:31:16.312[39m [32mINFO[39m [1mZ-WAVE[22m: Setting user callbacks

[90m2025-01-25 10:31:16.406[39m [34mDEBUG[39m [1mSOCKET[22m: Event INITED emitted to 8USlO1dysfMGKQoXAABX

[90m2025-01-25 10:31:16.595[39m [34mDEBUG[39m [1mSOCKET[22m: User disconnected from 8USlO1dysfMGKQoXAABX: transport close

Anyhow, I hope this helps identify the root cause of the problem.

@robertsLando
Copy link
Member

@AlCalzone any clue about any changes that could cause this starting from https://github.com/zwave-js/node-zwave-js/releases/tag/v14.3.7 ?

@AlCalzone
Copy link
Member

AlCalzone commented Jan 27, 2025 •
edited
Loading

It can only be this change then:
zwave-js/node-zwave-js@6a0d95c

Classes that emit events are now based on the DOM compatible EventTarget class instead of Node.js's proprietary EventEmitter. This means that some methods like prependListener no longer exist, but we haven't found any usage of this in the wild.

Maybe the MQTT portion is doing something with the event emitter I haven't found during my search?

Because I haven't noticed any issues, but I'm also not using MQTT.

robertsLando added a commit that referenced this issue Jan 27, 2025
@robertsLando
fix: use uncaughtException to catch missing exceptions
4905e0c 
Ref #4098
@robertsLando
Copy link
Member

Maybe the MQTT portion is doing something with the event emitter I haven't found during my search?

The strange thing is it doesn't crash immediately but after some days so no clue, never got any unexpected close on my side and I'm using mqtt as well.

@kpanchen @codemannh are you using mqtt or do you have mqtt gateway disabled?

I will release a new version now that should catch more exceptions so maybe we get some insight of the issue when it happens again.

@AlCalzone
Copy link
Member

These logs show crashes after just a few seconds:
#4098 (comment)

@codemannh
Copy link

Maybe the MQTT portion is doing something with the event emitter I haven't found during my search?

The strange thing is it doesn't crash immediately but after some days so no clue, never got any unexpected close on my side and I'm using mqtt as well.

@kpanchen @codemannh are you using mqtt or do you have mqtt gateway disabled?

I will release a new version now that should catch more exceptions so maybe we get some insight of the issue when it happens again.

I have mqtt disabled going straight into HomeAssistant. As one other person noted, I have a couple of devices that are a ways from my controller and it seems when they seem to take a long time updating status or controlling that is when the failure occurs. Sometimes it happens when I'm not looking directly at it, so without log information, its hard to say that is the case all the time.

@kpanchen
Copy link
Author

Maybe the MQTT portion is doing something with the event emitter I haven't found during my search?

The strange thing is it doesn't crash immediately but after some days so no clue, never got any unexpected close on my side and I'm using mqtt as well.

@kpanchen @codemannh are you using mqtt or do you have mqtt gateway disabled?

I will release a new version now that should catch more exceptions so maybe we get some insight of the issue when it happens again.

Yes, I'm using MQTT with OpenHab. My setup is very straightforward, everything is on the same box, Aeotec Z-Wave Z-Stick 7 ZWA010, Ubuntu latest release / patches, stand alone package for zwave-js-ui, Mosquito, OpenHab binding for MQTT. Everything was rock stable before that release.

I'll try a new release, let you know.

K.

@robertsLando
Copy link
Member

Could you guys try with 9.30.0 please and send me a log when the issue happens? We may have some more details about the error now

@codemannh
Copy link

Updated my environment to run 9.30
zwave-js-ui: 9.30.1.46a9eda
zwave-js: 14.3.8

@kpanchen
Copy link
Author

Updated to 9.30.1, will report!

Thank you very much for all the hard work!!

@robertsLando
Copy link
Member

Ok LMK :)

@kpanchen
Copy link
Author

kpanchen commented Feb 1, 2025

Just a quick update, don't want to speak too soon but 9.30.1 is running fine since updated 4 days ago. I've attached logs from today just in case you want to see if anything was caught in that Exception you added. Will keep monitoring.

20250201.zip

Thank you,
K.

@codemannh
Copy link

Unforutnately not so lucky here. My monitor identified zwavejsui crashed overnight. I still see nothing in the logs that were written by zwavejsui other than device related entries and they just stopped at the time of the crash.

@robertsLando
Copy link
Member

@AlCalzone In @kpanchen I see many nonce errors. No clue if that could highlight something

@codemannh What's the exact time the application crashed?

@codemannh
Copy link

codemannh commented Feb 3, 2025 via email

@robertsLando
Copy link
Member

@codemannh I have a feel so the issue is related to the log rotation. Do you see a log like Cleaning up log files... a bit before the crash in your logs? It could be also you don't see it as that is not written to the file in time before the crash happens

@codemannh
Copy link

Tried to reply to the email. I was incorrect in the statement I made there. The line Cleaning up log files is in the log at 00:44:00. There does not appear to be any issues though with the logs.

@robertsLando
Copy link
Member

robertsLando commented Feb 4, 2025 •
edited
Loading

As a test could you try please to disable ZUI log to file from settings and keep enabled only zwavejs one? Also giving that you are using docker could you share your docker-compose file?

@kpanchen
Copy link
Author

kpanchen commented Feb 6, 2025

Mine seems to work fine, don't see any obvious abnormalities now. Didn't have a time to have a look at the logs, let me know if you need/want me to test something else or provide more information.

K.

@robertsLando
Copy link
Member

@kpanchen thanks for your feedback! If it works maybe the issue is fixed and the issue faced by @codemannh is related to something else

@codemannh
Copy link

Just crashed again. Today it was at 11:50AM local time. Is there something specific I should look at to identify the cause?

@robertsLando
Copy link
Member

Maybe you could find some more informations on docker logs

@codemannh
Copy link

When the container crashes all the logs are gone. That was why I turned on the log to file. I've been actively monitoring the logs since yesterday, not seeing anything unusual. All entries are device related as well as my Uptime Kuma http ping to see if the container is up and running.

@robertsLando
Copy link
Member

robertsLando commented Feb 7, 2025 •
edited
Loading

@codemannh I suggest to enable json logging in docker compose, it could be that the exception for some reason is not picked up so will never be available in zui logs. See https://docs.docker.com/engine/logging/configure/#supported-logging-drivers and https://www.squadcast.com/blog/docker-compose-logs#storing-docker-compose-logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@robertsLando @AlCalzone @kpanchen @Matthew-MBG @codemannh