Should Wasmtime/Cranelift has an LTS release?

[alexcrichton]

In the Wasmtime meeting today we discussed #10074 and some related questions about MSRV/security patches. Wasmtime's current release process guarantees security fixes for 2 releases of Wasmtime, the current and previous release. This is not a large window and there will always be users who prefer stability/unchanging foundations rather than bumping the Wasmtime dependency each month.

One of the chief points brought up in the meeting would be to possible have an LTS release for Wasmtime. @fitzgen proposed perhaps every 10th version of Wasmtime as something easy to remember. I believe the roughly-agreed-upon definition of an LTS would look something like:

• The MSRV of the release in question would not change over its lifetime, even for security updates.
• Security fixes are guaranteed to be backported/released to releases in question.
• Bug fixes will be provided if we're explicitly requested to do it.

On the last point I don't think we can reasonably go through all bug fixes and proactively see if they're affecting older releases myself, but others may disagree!

I wanted to open this issue to have discussion of: should we do this? What are the exact/parameters/definition of an LTS we'd want?

---

Personally my thoughts are that I think something like this could make sense. One requirement I think we would want though is that LTS windows for releases would overlap slightly. For example of 20 is an LTS for only 10 releases then it means that as soon as 30 is released then 20 is no longer supported. That means there's no window for updating older LTS releases to a new LTS release where the old one is still supported. For me I'd probably say that we should overlap LTS windows by 2 release, meaning they're 2 months wide.

If we did that overlap then it would mean that any particular LTS release would be supported for 1 year. The last two months of that year users would be encouraged to update to the next LTS. This wouldn't mean though that we would always produce an LTS release once a year because LTS itself would be produced once every 10 months.

Personally I don't know how much effort it would be to maintain a year-old version of Wasmtime. Naively it doesn't seem like too too much work, but I would acknowledge there's a lot of unknown unknowns here. One of my chief concerns would be about keeping release automation up-to-date. Namely we'd probably want some sort of automated process which forces us to check in on LTS automation once a month or something like that to ensure it still works. Apart from that though 10 doesn't seem like an unreasonable window to support things.

If we do agree on LTS and every-10-versions there's also the nice coincidence that Wasmtime 30 will be branched in a little less than a week...

[alexcrichton]

Concretely what I'd envision is something like this, where each bar represents the duration that each release is receiving security updates:

gantt
    title A Gantt Diagram
    dateFormat  YYYY-MM-DD
    section lts1
    30.0.0           :a1, 2025-02-20, 365d
    31.0.0           :a2, 2025-03-20, 60d
    32.0.0           :a2, 2025-04-20, 60d
    33.0.0           :a2, 2025-05-20, 60d
    34.0.0           :a2, 2025-06-20, 60d
    35.0.0           :a2, 2025-07-20, 60d
    36.0.0           :a2, 2025-08-20, 60d
    37.0.0           :a2, 2025-09-20, 60d
    38.0.0           :a2, 2025-10-20, 60d
    39.0.0           :a2, 2025-11-20, 60d
    section lts2
    40.0.0           :a1, 2025-12-20, 365d
    41.0.0           :a2, 2026-01-20, 60d
    42.0.0           :a2, 2026-02-20, 60d
    43.0.0           :a2, 2026-03-20, 60d
    44.0.0           :a2, 2026-04-20, 60d
    45.0.0           :a2, 2026-05-20, 60d
    46.0.0           :a2, 2026-06-20, 60d
    47.0.0           :a2, 2026-07-20, 60d
    48.0.0           :a2, 2026-08-20, 60d
    49.0.0           :a2, 2026-09-20, 60d

Loading

[alexcrichton]

In thinking more about this as well, one of the things I want to be sure we consider is one of the major downsides of an LTS release: they infrequency of a release. I realize that's also the benefit so this can be counterintuitive. One advantage of the train release model is that there's never a rush to include a feature, if you miss a train the next one's coming soon. Personally I place quite a lot of weight on this advantage as it (a) makes landing features have way less pressure which and (b) encourages thoroughly implementing/reviewing/vetting features as opposed to rushing them in before a deadline. If LTS releases are widely used, then we run the risk of negating this advantage of the train release model where folks will feel pressured to rush something in last second for the 40.0.0 release for example because they're stuck with that feature set for a year.

To me this is in some sense a fundamental tradeoff between infrequent releases with long security updates and the train release model. While I can hope that by mixing both we strike a reasonable balance I still believe that this is going to bite us at some point. Basically this is something I think we need to acknowledge rather than pretending we can solve it one way or another.

[alexcrichton]

Also as a point of comparison Node.js has pretty good documentation on their release process which has a few differences:

• At any one point in time three LTS releases are supported (as opposed to one on average as proposed above)
• LTSes receive one year of feature/security updates, the one year of only security updates

I think the time scales involved for Node.js are much longer than something we should consider given our size/resourcing, but it's an data point to consider nonetheless.

[cfallin]

To me this is in some sense a fundamental tradeoff between infrequent releases with long security updates and the train release model.

I guess this fundamental tradeoff is the reason we have both kinds of Linux distributions, rolling/bleeding-edge and LTS. Different users/embedders can invest different amounts of time and get different "newest and greatest features" levels as a result. The way that I'm seeing this whole discussion is that we're realizing that the low end of this spectrum exists -- we can either choose to provide something for them (then Wasmtime may become and remain relevant beyond the bleeding-edge developers-and-also-users that we represent) or not (which is our perogative as OSS).

I also share your concerns about the time commitment this could imply -- so I think it's important to draw strong boundaries and say, e.g., only CVEs and nothing else are backported. If that's the case, taking 2023 and 2024 as representative, we had 8 CVEs. Three (1, 2, 3) were Cranelift miscompilations that I see as trivial to backport: fixes are typically removing a lowering rule or fixing a constant, and either are still applicable or not, likely won't have to be rewritten or rebased in tricky ways. One (1) is a logic bug in Windows filename sandboxing that again should be a simple point-fix to some logic to backport. The remaining four (1, 2, 3, 4) are various runtime bugs (two in GC, one in tail calls and trampolines, one in the way mut borrows are used); all of those kinds of issues could be tricky to rebase if we've done a lot of refactoring in the relevant area in the meantime. I guess what I'm taking away here, at least, is that most CVE fixes are point fixes to some localized logic and so on average the backport cost may be low, but there's no strict guarantee.

I think the rest is kind of a value judgment; I see this kind of thing as valuable personally as long as it doesn't cost us too much, and is more and more relevant as we see our software embedded more deeply into mainstream use-cases. Becoming one of the default answers when choosing a production Wasm engine is also, broadly speaking, something that aligns with all of our (paid contributors') interests even if the immediate users that would consume an LTS aren't directly paying for or contributing to help with the additional workload. Just my take...

[alexcrichton]

Personally I agree that we should probably do an LTS of some kind, but I also personally have no prior experience in either designing what an LTS is or actually implementing such a process. I don't disagree that it seems like it won't be too too much work, but at the same time I also want to try to be prepared for issues that may arise. What I'm thinking so far is:

• One issue is definitely infrastructure. Getting year-old CI infrastructure to work is not fun. Keeping LTSes low-maintenance for us I think is going to require periodic checkups on the CI of the LTS branch to ensure it's still working.
• Time pressure as well is not fun. I don't think we can have any sort of policy to fix this though. Instead I think we need to try to all have a handshake amongst each other that we'll try extra hard to not rush things in before an LTS release.
• Backporting into year-old code is not fun. At the same time though I'm not sure if we want a hard-and-fast rule that zero fixes other than security fixes are backported. What I might say though is that we should (a) guarantee security fixes, (b) support reviewing backporting bug fixes, and (c) refuse the backport new features. My hope is that this would strike a reasonable balance for us (only security fixes, never need to review backporting features) while still helping users who are "stuck" on an LTS if they encounter a bug and do their own backport.

I guess I'm using "not fun" as "this is a risk of greatly increasing maintenance cost on ourselves" above. Overall I'm mostly trying to think ahead and see what else is the cost of an LTS other than backporting security fixes. Backporting is a known quantity that I also don't have any issues with, but it's the unknown quantities that I'm wary of myself.

[fitzgen]

Agreed that supporting an LTS is probably the way to go. I think it signals project maturity, stability, and reliability, which should help us gain even more users. I suspect this will particularly sway potential "big" users that, once invested in Wasmtime, will be fully committed and will start sending upstream contributions. And I think users will in general feel better about using Wasmtime if there is an LTS, even if they don't use it. This is all a gamble in the sense that none of us know how much work supporting LTSes will be, but I think it is worth that gamble.

Regarding overlapping window of support on LTSes: yes we will definitely need to do that. Exact width of window I am unsure of. Two months might not be enough time, but is three? I don't know. I think ideally, from a user POV, two LTS would be supported at all times, so you'd have until the next LTS release to update from your old LTS release. But that is almost definitely more of a burden than we want to commit to from a maintainers POV.

Regarding exactly what gets backported and MSRVs:

• Agreed that we maintain the LTS's original MSRV
• Agreed we backport security fixes
• Agreed on never backporting features
• Regarding backporting bug fixes: I think this is a "patches welcome" situation, where we agree to review, but make no guarantees we will do anything beyond that ourselves.

Regarding CI infrastructure: at minimum we can schedule a full CI job for the LTS branch(es) to run ~weekly or something. Not perfect, but at least it is something.

Regarding pressure to land features before an LTS release: I think we just have to accept this. In some sense, any maintainer whose employer is not using an LTS doesn't need to worry about this, and if some company really needs a feature in a particular LTS release, it is on them to step up and do the work, or else simply wait for the next LTS release.

[cfallin]

it is on them to step up and do the work, or else simply wait for the next LTS release.

Or maintain an internal fork -- that is always an option when an organization really wants Wasmtime to work a certain way and has the resources to make it so. An LTS still benefits such folks because it gives them a stable base to start from (then hopefully they can upstream their work in due time).