Help understanding NVM versions

[billmoseley]

I have two ZST39s and I'm trying to copy the NVM from one to the other.

@AlCalzone noted zwave-js "converts the NVM to the target format," but I'm unclear what "target" means there. Is that the SDK (protocol?) version?

I'm confused because wouldn't upgrading the firmware mean that the NVM would need to be upgraded to match?

My source ZST39 has v7.18.1 which is too old for ZUI to backup its NVM. I used PC Controller to sucessfully back up the NVM. Then I attempted using PC Controller to restore that NVM onto a ZST39 with v7.22.1 -- and it bricked it and cannot be repaired.

My question is can ZUI "convert" the NVM from the v7.18.1 and write it to a v7.22.1 ZST39?

I converted the .hex v7.18.1 to .bin and the nvmtool reports protocol version 7.18.1

{
  "format": 4,
  "controller": {
    "protocolVersion": "7.18.1",
    "applicationVersion": "1.1.0",
    "homeId": "0xd20ebe47",
    "nodeId": 1,

Before attempting to restore onto the newer ZST39 its NVM showed:

{
  "format": 5,
  "controller": {
    "protocolVersion": "7.22.0",
    "applicationVersion": "7.34.0",
    "homeId": "0xd997a6b8",
    "nodeId": 1,

[AlCalzone]

I'm unclear what "target" means there. Is that the SDK (protocol?) version?

There is a difference between 500, 700 and 800 series NVMs, although 700 and 800 series are very similar. Those need to be handled before restoring, because otherwise the controller won't be able to read the NVM contents.

There are also subtle differences between SDK versions. Most of those should be handled by the firmware automatically, but it doesn't always work, and only when the SDK version is newer than the one the NVM was copied from.

One of the less subtle ones is that the 800 series NVM size has been reduced between 7.18 and 7.19, so the contents have to be "packed" differently. The firmware can't do that.

PC controller does not do any conversions. It blindly copies the NVM data and in your case you ended up with an NVM that was missing the last 8 KB.

Since Z-Wave JS was able to parse the 7.18.1 NVM, it would have been able to restore it onto the 7.22 controller without bricking it.

---

TL;DR: "target" is the combination of the Z-Wave series (500, 700, 800) and SDK version of the controller you're trying to restore onto.

[billmoseley]

One of the less subtle ones is that the 800 series NVM size has been reduced between 7.18 and 7.19, so the contents have to be "packed" differently. The firmware can't do that.

Interesting. I can see where zwave-js-ui could convert the NVM to match the target device and SDK. But that sentence sounds like trying to update the firmware on the zst39 from 7.18 to 1.19 (or the latest 7.22.1) would require that the NVM is also updated.

My guess is zwave-js-ui doesn't also update the NVM on a firmware update. Is that correct?

This may be wrong, but I imagine the boot loader is like the MBR on a PC, the firmware like the OS, and NVM is a data region. I assume I bricked my zst39 hard, meaning minicom isn't able to see any response from the device, is because PC Controller somehow corrupted the boot loader code.

Thank you @AlCalzone for the explanation. It really helps to understand these details.

[AlCalzone]

The Z-Wave firmwares have a routine on startup that checks whether the NVM (data region) was written with an older firmware and requires a migration. If that migration is necessary, it performs the migration (e.g. store the radio configuration in a different format). This is meant for forwards-compatibility.

The base firmware included in the SDK has some checks for unsupported situations (like a truly incompatible NVM layout, or missing information), where it unfortunately just crashes and becomes unresponsive instead of returning to the bootloader or writing defaults for the missing info. AFAIK the Zooz firmware doesn't change that (I've seen reports before), and I suppose you've triggered that with your NVM restore.

zwave-js-ui doesn't also update the NVM on a firmware update. Is that correct?

Correct. We trust the firmware to do the necessary changes. I'm not sure if this reduction in NVM size is handled automatically, but with the restore using PC Controller you most likely skipped it since the newer firmware doesn't let you write the full contents of the old NVM.

If you have a J-Link (or a compatible programmer), you may be able to change the NVM contents and possibly restore operation that way. It is pretty involved though and might not be worth the cost (several hundred $ for an original, around 20-30$ for a china clone).

[billmoseley]

My replacement ZST 39 LR arrived a day early. Everything went perfectly:

1. I restored my "production" NVM v7.18.1 onto the brand new ZST 39 that came with v7.22.1
2. I upgraded the firmware on my v7.18.1 ZST 39 to v7.22.1

I did all that on a separate machine running ZUI in docker w/o Home Assistant.

I now have two ZST 39s running v7.22.1 with the same NVM and both see and control all my nodes just fine.

Thanks for all the help! This is really great software.

A few minor questions came up while doing this:

1. I assumed I didn't want two controllers on at the same time on the same network. Is it enough to shut down zwave-js-ui? The USB stick is still powered up (the blue light in the ZST39 case is still illuminated). Or is it important to unplug the USB stick?
2. I'm curious why nodes.json doesn't have the home ID prefix, like the .jsonl files have. Since it's mapped by node ID I would think it would be specific to a given NVM (with its Home ID).

[AlCalzone]

1. I'd unplug the 2nd one. It may still ACK incoming frames, which may mess with the traffic flow.
2. The .jsonl files come from node-zwave-js, while the nodes.json is created by Z-Wave JS UI. Since very recently, it contains its data grouped by home ID for the reason you mentioned.