Are Minio/non-AWS buckets supposed to be supported?

[onpaws]

I'm trying to use aws-sdk-ruby with bucket mybucket hosted at a non-AWS provider.
While I can ls the bucket via the aws s3 cli OK:

$ aws s3 ls mybucket --profile ovh --endpoint-url https://storage.gra.cloud.ovh.net/
2022-09-06 15:19:09     992388 foo.jpg

The following Ruby script returns a Aws::S3::Errors::Http405Error:

require 'aws-sdk-s3'

options = {}
options[:access_key_id] = "SNIP"
options[:secret_access_key] = "SNIP"
options[:endpoint] = "https://storage.gra.cloud.ovh.net"
options[:region] = "gra"

s3_client = Aws::S3::Client.new(options)
@s3 = Aws::S3::Resource.new(client: s3_client)
@bucket = @s3.bucket('mybucket')
print('exists?', @bucket.exists?)

I'm an aws-sdk-ruby noob but basically the reason I thought custom S3 endpoints might be possible is because 1) the provider claims S3 compat and 2) I noticed @awood45 's old comment.

When debugging and stepping through, I was mildly surprised to see the Host header:

I'm not sure how subdomain aspects are supposed to work, but it seems mildly sus.
My goal is for @bucket.exists? to return true.
Is there something I'm doing wrong? Other config options I might consider?
Or are custom S3 endpoints not a thing?

Contents of ~/.aws/*

~/.aws/config

[profile ovh]
region=gra
s3 =
  endpoint_url = https://storage.gra.cloud.ovh.net/
  signature_version = s3v4
s3api =
  endpoint_url = https://storage.gra.cloud.ovh.net/

~/.aws/credentials

[ovh]
aws_secret_access_key = snip
aws_access_key_id = snip          

---

aws-sdk-ruby v1.114.0
Ruby v2.7.0
macOS 12.5.1 on x86_64
API (I believe) OpenStack https://docs.openstack.org/api-ref/object-store/
Provider OVH https://www.ovhcloud.com/en/public-cloud/object-storage/

[mullermp]

Yes, in theory those are supported but those use cases are not well understood or tested from our side admittedly. It is correct that the S3 bucket should be prepended to the URL as a subdomain (it's called a virtual host style). This is S3's preferred style. Buckets that aren't DNS compliant are in the URL path. I am thinking that your custom endpoint expects path style, such as /bucket/key, and I also believe this to be a difference in default behavior between the CLI and most of the SDKs. Try configuring either force_path_style: true (preferred) or disable_host_prefix_injection: true on your S3 client and it should not prepend.

[onpaws]

TIL about force_path_style, got it! Thanks for the great explanation.

(I also just saw Jeff's post which provides more context.)

Happily, when I include options[:force_path_style] = "true" the script now works as expected. Hooray! 🎉

I'll go ahead and close this issue. EDIT: nvm, apparently GH 'discussions' don't get closed. All good.
Thanks again

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.