Mounting fails when using MinIO and OIDC authentication

[2022tgoel]

Mountpoint for Amazon S3 version

mount-s3 1.13.0

AWS Region

No response

Describe the running environment

I am mounting a directory stored in the MinIO S3-compatible storage service running locally in a docker container.

I am setting AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_SESSION_NAME, and AWS_ROLE_ARN to use OIDC for authentication.

I have verified that the credentials I am providing are valid through:

aws sts assume-role-with-web-identity \
  --role-arn $AWS_ROLE_ARN \
  --role-session-name $AWS_ROLE_SESSION_NAME \
  --web-identity-token "$(cat $AWS_WEB_IDENTITY_TOKEN_FILE)"  \
  --endpoint-url http://127.0.0.1:19080

The command above succeeds.

Mountpoint options

S3_ENDPOINT_URL=http://127.0.0.1:19080 mount-s3 blobnet-00 test-mount-3 --debug-crt --log-directory logs

What happened?

Error: Failed to create S3 client

Caused by:
    0: initial ListObjectsV2 failed for bucket blobnet-00 in region us-east-1
    1: Client error
    2: No signing credentials available, see CRT debug logs
Error: Failed to create mount process

CRT debug logs:
mountpoint-s3-2024-12-17T17-09-47Z.4h5iTP.log

It claims that the STS web identity path responds with 400 bad request. However, I don't understand this for two reasons

1. It works with aws sts
2. I got a wireshark capture of the exchange between mount-s3 and minio, and responds with 200 OK and an access key, indicating that everything is working. I can share the PCAP, for some reason github isn't letting me attach it.

Relevant log output

mountpoint-s3-2024-12-17T17-09-47Z.4h5iTP.log

[2022tgoel]

@pawalt

[unexge]

Hey @2022tgoel, the S3_ENDPOINT_URL environment variable only affects S3's endpoint and not the credential provider's endpoint. I can see from your logs that Mountpoint still tries to make a request to sts.us-east-1.amazonaws.com to fetch temporary credentials and then gets 400 response back.

Not sure if it'd be viable for you, and also I haven't tested myself but using aws sts assume-role-with-web-identity as a process credential provider might work – you probably need to massage output of your command to match the desired output.

[2022tgoel]

Hi, thanks that makes sense.
Unfortunately I am trying to test whether the Assume role provider works with mount-s3, so using the process credentials provider will defeat my purpose.
I think the issue is that I should be able to use AWS_ENDPOINT_URL_STS to set the endpoint for STS, but that does not work due to mountpoint's usage of aws-c-auth for credentials, which only supports sts.amazonaws.com. Would it be possible to switch to https://github.com/awslabs/aws-sdk-rust/ or botocore? I can also make a contribution for this since I would like to get it working soon.

[unexge]

Hey @2022tgoel, as you noted Mountpoint uses aws-c-auth under the hood for credentials and such a change/contribution should be done on that side. Feel free to create an issue with them to discuss if they would be open for a contribution of such a change.

I'll also discuss with my team regarding accepting a contribution to change credential provider to AWS Rust SDK rather than CRT, but that's probably a bigger change and would require some discussions.

[pawalt]

@unexge is there anyone that can bump the linked issue in aws-c-auth? It looks like those issues don't get a lot of traffic.

[muddyfish]

We've reached out to the team about the issue, but don't have anything to share yet. (FYI @waahm7)

[waahm7]

Thanks, This issue is fixed in https://github.com/awslabs/aws-c-auth/releases/tag/v0.8.6 and now CRT will respect the environment variable.

[pawalt]

@muddyfish I assume this is a matter of bumping aws-c-auth in the next mountpoint release? We may just bump it on our fork depending on backwards compatibility or wait for y'all to cut a release.

[unexge]

Hey @pawalt, we bumped CRT submodules and the fix will be included in the next Mountpoint release.

[pawalt]

@unexge thanks so much :) we're looking forward to pulling this in