Add optional tagType query param to tags list endpoint

[jdolitsky]

Adds the ability to do a server-side filter of "digest" tags (e.g. sha256-...).

This is done by creating a tagType classification, which can be used to filter things. This works similar to the artifactType filter on the referrers API.

This is intended to exclude or isolate tags which are considered "non-normal". If another type of common non-normal tag appears in the wild, it can be added as a new tagType. In general, this is considered an anti-pattern and clients are encouraged to use the subject field and referrers API etc. to connect objects in the registry.

[tianon]

"this is considered an anti-pattern" seems like a pretty good reason not to add even more functionality supporting this in the spec, right? Especially given the long tail of adoption?

[jdolitsky]

"this is considered an anti-pattern" seems like a pretty good reason not to add even more functionality supporting this in the spec, right? Especially given the long tail of adoption?

That's a fair point. However, specifically in the case of tags falling into this "digest" category, we're only seeing the number of these tags grow vs. slow down/migrating to use referrers.

[jdolitsky]

From discussion, alternative:

• Request tag endpoint with Accept: <new_media_type>
• If Content-Type of response has this mediatype, parse it new way
• Query params for search query regex and artifact type etc.
• Support for ETAG header

[jcarter3]

If implementers are resource constrained on adding referrers, it seems very unlikely for something like this to get implemented either.

FWIW, Docker Hub does have referrers on the roadmap, and would not likely prioritize this ahead of that.

[jcarter3]

What about cosign style tags with a .sig or .att suffix? Are those considered "digest" style? They aren't valid digests, nor referrers, but are "not real".

[jdolitsky]

If implementers are resource constrained on adding referrers, it seems very unlikely for something like this to get implemented either.

Fair point. I do think that depends. There was further discussion on the call today that there are other use cases for filtering tags unrelated to referrers-related behaviour:

• Filtering out non-semver
• Filtering out semver rc tags
• "Give me only things pushed in last day" (not currently possible in current spec)

What about cosign style tags with a .sig or .att suffix? Are those considered "digest" style? They aren't valid digests, nor referrers, but are "not real".

Yes, the .att / .sig tags are intended to fall into "digest" the category. As currently written in this PR, it applies to any tag prefixed with the string "sha256-" (we could also have it apply to "sha512-" and "blake-" if necessary). A regex filter is another possibility vs. known tag types.

[jcarter3]

I agree that other types of tag listing would be useful, but that seems much greater in scope. Current spec limits the response to be a string list of tags, which severely limits the usefulness. If we want to do date filters, it'd be more useful to be able to return some metadata with the tag, which would be a new endpoint (or add new, optional fields in the response to keep backwards compatibility)

[jdolitsky]

@jcarter3 - see this comment above for @sudo-bmitch rough idea for a v2 tag listing endpoint: #568 (comment)

[sudo-bmitch]

My gut reaction on this is that it's unlikely to solve the target issue, since registries that don't implement referrers are also unlikely to implement new tag listing features. But my bigger goals include a v2 of the tag listing API, perhaps merged with a manifest listing described in #222. Some of my thoughts were already captured above, but to put them all in one place:

• Switch between the v1 and v2 API with clients sending something like Accept: application/vnd.oci.tags.v2+json, and the server can respond with the appropriate Content-Type header for clients to detect what response the server is sending.
• Include full descriptors for each tag.
• Registries may include additional metadata, e.g. pull counts, upload time, uploading user, if referrers exist, etc.
• Registries can also include a history for a tag, previous descriptors that the tag has pointed to, allowing clients to implement an "undo" feature.
• Allow an improved query interface to select tags by name, mediaType, artifactType, upload date, etc.
• Allow alternate sort orders, e.g. most recent push, most pulls, and semver.
• Support ETAG headers described in Proposal: Document ETag and Precondition Semantics #250.

These ideas come from the different types of questions people are solving when querying the tag listing:

• What is the most recent semver version of this image (i.e. how to make Watchtower like tools have a lower impact)?
• Have any new tags been pushed since the last query (facilitate mirroring tools)?
• What are the variations of a given version available, e.g. scratch, alpine, and libc variants of an image?
• List all content of a particular artifactType, all helm charts or docker compose yamls.
• List all tags matching a particular expression to find all versions of a particular image variant.
• What tag previously pointed to a given digest?
• What was the previous digest for a given tag (where's the undo button)?
• What content can be manually deleted? (Low pull counts, untagged manifests, digest tag schema pointing to a digest that doesn't exist, etc.)

Support for some of these features could dramatically lower the load on registries, giving them motivation to adopt a v2. Particularly since I suspect a significant percentage of tag listing APIs are from automated tools checking to see when a new tag has been pushed, either to mirror the tag, update builds/deployments, or launch vulnerability scanners.

All of the above is probably a big enough change to warrant a working group, but I don't have the bandwidth to run another one of those right now (particularly since two of the WGs I'm on are currently paused pending maintainer bandwidth).