Warg development roadmap proposal

[lann]

Based on a number of discussions about the best way to get Warg into production, I'd like to propose the following development roadmap:

Existing implementation

The existing implementation will become "the Warg prototype"; code will be ported from the prototype into the new codebase as appropriate.

New implementation

The new implementation will use a phased approach to deliver a minimal subset of Warg's functionality as soon as possible with additional functionality being added in incremental updates while maintaining a high level of quality and stability for consumers.

Phase 1: Warg-on-OCI

Design proposal: https://hackmd.io/pAO6nGgjR9K22MR9mNWbNw
PoC: https://github.com/lann/warg-loader

This iteration will scale Warg back to a fairly basic versioned package artifact delivery system that requires only an OCI Distribution Spec-compliant registry as infrastructure. The Warg-specific logic will be implemented by a Rust library that exposes a simple interface which can: resolve namespaces to hosts, list versions of a named package, and fetch content for a package version. This should be sufficient to allow downstream consumers like wasm-compose to start using Warg. Package publishing will be via existing OCI tooling like ORAS.

Note: Below this point hasn't been broadly discussed; this represents one possible path toward
the functionality currently implemented in this repo.

Phase 2: Package Logs

This iteration will add Warg's immutable package logs to the OCI implementation by publishing the full log (on each update) to a specific OCI tag, e.g. <registry>/<namespace>/<package>:warg-log. At this phase the log will include just the "publish" and "yank" entries, enabling yanks to be exposed in the Rust library. This will also introduce Warg publishing tooling which will still largely wrap an existing OCI implementation but with the addition of log management. One notable omission at this phase is publisher record signing; the registry operator will still need to be fully trusted at this point.

Phase 3: Registry Logs (Package Transparency)

This phase will add the mechanisms necessary to audit the operation of a registry via verifiable logs covering the state of the registry over time. This will be the first phase where custom infrastructure may be required, though we may want to re-evaluate e.g. Sigstore's Rekor for this.

Further Features

At this point there will be several features that should be available to implement in any order or in parallel based on consumer feedback and interest:

• Federation
• Package publisher log record signing & ACL
• Registry Indexing & UI

[esoterra]

Phase 2 Publish Concern

I'm not clear in Phase 2 what the OCI-based publish flow would look like. Is the client authenticating with OCI and attempting to publish the new full log to the tag? If so, how does a registry enforce policy on the upload? Is there a way to interject scanning/policy into commodity OCI registries?

[lann]

Phase 2 would still have no server-side logic; I just think "yank" is important enough to be a quick follow-on to the bare bones fetch operations.

[esoterra]

I'm just not clear what the value of having the log is if the registry isn't doing any kind of validation on the log and it can be updated in one step to a completely different log?

What advantage does that have over just putting the entire package state as the value of the tag including the current state of every release?

[lann]

The validity of an immutable log would still be verifiable by individual consumers over time, you just wouldn't (yet) have the "network effect" verifiability afforded by transparency logs.

[esoterra]

I'm not sure in this case what's different between verifying that the state changed in a safe way (e.g. additively) and that this whole new log shares history with the whole log I downloaded last time.

[sajayantony]

Potentially you could attach the log to the component which may be obtained independently and the logs or state change can be moved along to another registry as well if this can become a reference attachment.

[devigned]

Some of the greatest value of a component registry, in my option, is the ability to query for a component that implements a given interface. For example, as an application developer that needs to use a key/value store, I would like to ask for each of the components known to the registry that implement the key/value interface. Has there been thought about querying for components that may implement a given interface, and where that might fit into the phases?

[lann]

That would fall under "Registry Indexing".

[devigned]

This feature seems like it would be core to the developer experience. I'd vote for querying / UI / UX to be only second to storing components. In my humble opinion, success of developers building applications with components should be top priority.

[lann]

Registry indexing in particular can be done in parallel with everything else (as indeed has been the case for the current implementation), so I think its just a question of resourcing at this point.

[devigned]

Sounds good. Just worried me not seeing that use case called out as a priority in the phases.

[thomastaylor312]

I'd really prefer if the indexing/search was called out as phase 2 (or at least as a named phase). I heartily concur with @devigned's assertion that this is what developers actually need. The log and transparency stuff is not what people are necessarily looking for and (as is called out) can be handled by existing tools like Rekor (or whatever else people are already using).

[calvinrp]

To recap yesterday's (Jan 3, 2024) Registry SIG meeting, Phase 1 is moving forward. The deliverable is a client library (that can be used within Cargo Component and other tools) that can pull (not publish) Wasm packages from an OCI registry. Tentatively, this library is called warg-loader and @lann is planning on building on his prototype and pull it in as a BA project. Expectations were set that would happen over the next few weeks.

We can delay getting agreement on what happens after Phase 1, for the moment. We expect there will be many more discussions on the topic. The hope is that we will have more clarity on the path forward on the other side of the BA Plumbers Summit at the end of the month (January 2024).

[squillace]

where does publish happen?

[calvinrp]

In Phase 1, through existing OCI tools. This is not intended as a "complete" or even a "good" developer experience.

Phase 1 is primarily to pull published WASI Preview 2 interfaces from OCI.

[lann]

Phase 1 publish will just use ORAS (or whatever OCI tooling one might be familiar with). I have a test-as-PoC here which I'll turn into documentation.

[jsturtevant]

Took a look at the design document for the OCI artifacts, it looks like the manifest format is still being determined? I am mostly interested with how the artifacts will be consumed by the various wasm and container runtimes.

I believe some folks from this group are aware/involved; In the CNCF wg-wasm group we have started pulling folks together from across various projects in the eco system to come up with an OCI Artifact format that will work for multiple wasm runtimes and container runtimes because as of now it is a bit fragmented and project specific.

Hope someone from Warg effort can join and we can coordinate with the WASM Artifacts format in Warg.

[lann]

I am still deeply confused about the status of artifact media types wrt opencontainers/image-spec#1131 🙈

[squillace]

opencontainers/image-spec#1141 has the most updated discussion; my take on it is that it's clear artifacts will be the way to go, the conversation now is on how to migrate that direction and update the artifacts spec such that older clients don't bonk. If you dig into that, it becomes clear to me, at least, that the remaining problem is the oddities in the spec wrt to tools that already exist and the clear assumption in OCI that it would always and only be container images.

That's my take, but have a read.

[lann]

Oh yes sorry, opencontainers/image-spec#1141 was the one I was trying to find. In particular the thread ending in opencontainers/image-spec#1141 (comment) seems like it covers a lot of important unresolved details.

[squillace]

I big part of this desire to use artifacts is that it lets us step outside of the OS-specific and arch-specific world of clients for "one thing you want to run"...... it'll get there.

[jsturtevant]

Here is a document that outlines the goals the wg-wasm is working on and starts to put together what a manifest might look like, hopefully after a few meetings we will have something more concrete: https://docs.google.com/document/d/1bZLjDpcG22PruvSUxGH-DNL906GEV0fYIbHQH7eNzqs/edit

We will have the first meeting next week Tuesday Jan 16th at 5:00 pm CET/ 8:00 am PST in the WG Wasm meeting. Hope to see some of you there!

[thomastaylor312]

Just wanted to call out here as well that as you integrate with OCI, let me know if anything is missing from the oci-distribution crate as I am one of the maintainers! Just want to make sure we don't block anything from that side

[lann]

I'll send PRs if I need anything! So far I think we're good for initial features.

[lann]

I have most of what is described as "Phase 1" code-complete here: https://github.com/lann/warg-loader

Included is a basic CLI for listing and fetching packages and a README with brief instructions for testing.

[lann]

With a bit more functionality added today, using an OCI host that differs from the registry domain now works:

$ curl https://lann.dev/.well-known/warg/registry.json
{
        "ociRegistry": "ghcr.io",
        "ociNamespacePrefix": "lann/"
}
$ warg-loader lann.dev test:pkg
Package: test:pkg
Versions:
  1.0.0

[esoterra]

While there's broad consensus on Phase 1, there's rightfully a lot of concern from SIG-Registries members about Phase 2+.

Core Observation

To start with, I would argue that the most fundamental parts of warg are well understood, essentially complete, and easily justified to stakeholders.

e.g.

• basic registry functionality
• transparency elements
  • data structures
  • proofs & verification
  • client-server protocol

Some other elements of warg however have never fully been solved, have lots of work left, and are not appealing to all of our stakeholders. These elements have slowed us down and made completion and stability harder to reach.

e.g.

• package delegation (in-log auth & user key management),
• log-based operator key management

Simplifying Warg

Instead of marking our Warg implementation as a "prototype", simplify it into something that can be ready and stable quickly and solves or avoids unsolved problems.

1. Remove package delegation
   • Make package records unsigned
   • Remove entries related to key management
2. Remove log-based operator key management
   • Make operator records unsigned
   • Remove entries related to key management
   • Rename "operator log" to "namespace log"
3. Integrate sigstore for operator key management

Changing Phase 2

Getting to and stabilizing this simpler version of warg can be our plan for Phase 2. There's no clear deadline for when "Phase 2" needs to happen anyway and if we don't have the resources to implement it, we can pivot to something simpler. So I think being more optimistic with Phase 2 doesn't cost us much.

[esoterra]

Warg-loader Release Criteria Proposal

Before we release/launch/announce warg-loader, I think it should be able to support more than one backing service ideally the existing warg client. This generally mirrors the idea in WASI of not announcing preview 2 till there are two worlds with a similar purpose: making it clear that WASI is not just cli and Warg is not just OCI.

[calvinrp]

The prototype and scope of the proposed warg-loader would be limited to read-only pull/fetch from OCI (and likely Warg and others) but lack publishing capability. There's also some questions on whether warg-loader would be just a library or also a CLI tool. That is being worked out.

Going to pose a question, do we want to extend the warg CLI and library to support OCI pull & push (publish)? This could make it easier for cargo-component / wit CLI (and others) to continue to support publishing and give OCI first-class support along side Warg. This would not preclude usage of existing OCI tools or the warg-loader development path. But it could make it easier to share dependency solve, bundling/unbundling, locking/unlocking, and other logic for use across Warg and OCI.

[lann]

warg already effectively has support for unauthenticated OCI pull "for free": the server can just return an httpGet content source pointing directly at the OCI blob.

Authenticated OCI pull and push is more complicated. The conventions around OCI auth are messy, being based largely on Docker auth. I think it would be fine for warg to support these but I don't have a good grasp on exactly what that should look like.

[calvinrp]

I should clarify, I was posing the question of whether the current warg CLI and client library should support pulling/pushing to OCI directly without requiring a Warg registry to be available.

[lann]

We certainly could but it would require major changes to the warg-client crate's API; for example, it is impossible to implement Client::download_content for an OCI backend (with that current function signature).

[calvinrp]

Related question, do we want first-class support for both OCI and Warg protocols? Or support OCI in a different way, for instance just for locked components that could be treated as deployment artifacts? Do we want to do dep solve for unlocked components across OCI registries (or a mix of OCI and Warg registries)?