Inconsistent interfaces for configuring docker registry for airgap install

[collinbachi]

A kots user can use one of several different methods to configure their docker registry during airgap installs. It is often unclear which methods are supported. Available methods also vary depending on the interface being used (kots admin-console push-images, kots install, and the admin console airgap install GUI are the three interfaces).

For the following examples, I'll use gcr.io/dev-project/subdir/ as an example docker registry path.

kubectl kots admin-console push-images

kubectl kots admin-console push-images ~/Downloads/kotsadm.tar.gz gcr.io/dev-project/subdir/ results in an error due to the trailing slash: failed to parse dest image name gcr.io/dev-project/subdir//kotsadm:v1.20.3: invalid reference format

kubectl kots admin-console push-images ~/Downloads/kotsadm.tar.gz gcr.io/dev-project/subdir works as expected: Pushing gcr.io/dev-project/subdir/kotsadm:v1.20.3

There is no configuration option to specify a registry "namespace" (in contrast with admin console airgap install GUI).

kubectl kots install

kubectl kots install example/stable --kotsadm-registry gcr.io/dev-project/subdir/ --airgap-bundle example.airgap

Here, the trailing slash is handled correctly when pushing the airgap images. It is not handled correctly when deploying the kotsadm images to the cluster, instead producing image references like: gcr.io/dev-project/subdir//kotsadm/minio:v1.21.0. Notice also that /kotsadm/ has been inserted, which is inconsistent with the behavior of admin-console push-images above.

The kotsadm-registry is used to both pull the kotsadm images, and as a target to push the images from the airgap bundle (in contrast with admin console airgap install GUI, where you can specify a different repo to push the airgap bundle images.)

admin console airgap install GUI

The airgap install GUI differs from the CLI options in several ways:

• unauthenticated docker registries are not supported (see Airgap install GUI doesn't support unauthenticated docker registries #1189)
• there is a separate "namespace" field
• airgap bundle images can be pushed to a different registry than kotsadm images

For the example registry, /dev-project/subdir can be added to the end of the "Hostname" field, or included separately in the "Namespace" field. (I did not take the time to validate whether trailing slashes are handled gracefully in either field.)

Expectations

My expectations as an end-user would be that docker registries can be configured the same way, regardless of which installation type I use. I would assume the same library is handing the input for each of them.

I am confused by the concept of the registry "namespace", and would prefer to provide a single URL.

[collinbachi]

kubectl upstream upgrade requires a --kotsadm-namespace, unlike the other two CLI commands that take a --kotsadm-registry. This makes it impossible to push the upgrade images to a docker registry without a namespace, as far as I can tell.

[travis-credoai]

We have several customers that wish to use the private registry features. I can vouch for the above inconsistency which makes for challenging customer support be it live, written or via documentation.

[travis-credoai]

@collinbachi I recently found

kots/pkg/docker/registry/auth.go

Line 82 in 5884615

if IsECREndpoint(endpoint) && username != "AWS" {

which I used with ECR and granting kubelet pull privileges via another mechanism (ie not IAM user credentials). Something similar could bypass setting authorization headers if username/password are empty.

Some customers may prefer not to grant even read privileges to kotsadm but simply configure it to substitute image registry & repo references. In these cases we don't want to be blocked by kotsadm not being able to auth to the registry and enumerate images.