(assets): docker login clashes with ecr-login

[SamStephens]

Describe the bug

This is a reopening of #14265, which was closed for inactivity. However, allowing usage of https://github.com/awslabs/amazon-ecr-credential-helper has become substantially more urgent because ECR have recently made a change to increase the length of tokens beyond the size that the Windows credential manager can store, 2500 characters. This is completely breaking for Windows users of Docker with ECR, and as amazon-ecr-credential-helper does not work with CDK and https://github.com/dougrday/docker-credential-plaintext did not work for me, Windows users are left with no option other than to code their own credential manager (or patch amazon-ecr-credential-helper to behave as though AWS_ECR_IGNORE_CREDS_STORAGE=true were set).

The only reason I'm not completely blocked by this is because I have an EC2 Ubuntu machine configured that I can use to do my CDK deploys.

See aws/aws-cli#5636 and aws/containers-roadmap#1589 for context.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Either to be able to use amazon-ecr-credential-helper with CDK, or for ECR tokens to comply with the 2,500 character limit Windows Credential Manager imposes.

Current Behavior

No way to use CDK Docker on Windows.

The actual issue I face is that when I try and use amazon-ecr-credential-helper, I need to set AWS_ECR_IGNORE_CREDS_STORAGE=true, as CDK expects to be able to use docker login with the credentials it receives. However when I call the CDK CLI with that environment variable set, I still see the Error saving credentials: error storing credentials - err: exit status 1, out: not implemented` error from Docker, indicating that however CDK invokes Docker, that environment variable is presumably not passed through.

Reproduction Steps

Perform any deploy involving Docker on Windows.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

n/a

Framework Version

No response

Node.js Version

n/a

OS

Windows and WSL

Language

Python, TypeScript, Java, .NET, Go

Language Version

No response

Other information

No response

[khushail]

@SamStephens , Thanks for reporting this.CDK Team is aware of the issue. Apparently the issue was caused by ECR team's code change and they are working on a fix right now. Will share more updates as we get them.

[gioruf-rslabs]

Hi,
This is affecting our CI/CD pipelines that, unfortunately, have to run on windows.
That said, we found an extremely weird workaround.
@SamStephens, if you pass the docker password using the stdin (even on powershell) it works:

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 123456789.dkr.ecr.eu-west-1.amazonaws.com

While

docker login --username AWS --password "LONG ECR TOKEN"

returns:

Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `The stub received bad data.``

As I said, extremely weird, but "Windows works in mysterious ways".

[gioruf-rslabs]

It gets better, the length of the token seems to be quite inconsistent:

Probably some of the nodes are still on the old, non-buggy version, or they are rolling out the fix as I am writing.

[jlouros]

Also getting the same issue on my CI pipelines and locally.

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 597747411843.dkr.ecr.eu-west-1.amazonaws.com throws Error saving credentials: error storing credentials - err: exit status 1, out: error storing credentials - err: exit status 1, out: The stub received bad data.``

[khushail]

Thanks for having patience. Using IAM User credentials when calling ECR's GetAuthorizationToken API results in a shorter token size, which may be a potential workaround. Mitigation is currently in progress.