Onelake incompatible with rustls

[djouallah]

Environment

Delta-rs version: 0.25.1

Binding: Python

Environment: Linux

• Cloud provider: Microsoft Fabric onelake
• OS:
• Other:

---

running

pip install deltalake --upgrade
dt = DeltaTable(result_table)

get

OSError: Generic MicrosoftAzure error: Error after 10 retries in 2.475340161s, max_retries:10, retry_timeout:180s, source:error sending request for url (https://onelake.blob.fabric.microsoft.com/sqlengines/ETL.Lakehouse/Tables/dbo/results/_delta_log/_last_checkpoint)

cc @RobinLin666

[ion-elgreco]

@djouallah have you tried another os to read with?

(btw a normal azure read works fine, manually verified that)

[ion-elgreco]

@djouallah I suggest you try to compile of each of these branches and report back if it resolves the connection issue:

• https://github.com/ion-elgreco/delta-rs/tree/fix/rustls
• https://github.com/ion-elgreco/delta-rs/tree/fix/use-native-tls-vendored

[djouallah]

windows works fine with onelake

[ion-elgreco]

Ok then it's related to the SSL handshakes. Please compile both of these branches I've shared above and let me know the results of each, then I can narrow down the fix

[ion-elgreco]

@djouallah can you check the latest release out please

[djouallah]

same errors

OSError: Generic MicrosoftAzure error: Error after 10 retries in 3.247428205s, max_retries:10, retry_timeout:180s, source:error sending request for url (https://onelake.blob.fabric.microsoft.com/sqlengines/ETL.Lakehouse/Tables/T10/streaming/_delta_log/_last_checkpoint)

[ion-elgreco]

@djouallah then it's a seperate issue for you, which architecture are you running it on?

[djouallah]

Linux , python notebook running in fabric

[ion-elgreco]

Linux , python notebook running in fabric

That's OS, I mean CPU architecture

[djouallah]

Intel 64

[liamphmurphy]

We're seeing similar errors on our S3 bucket on delta tables that were written on v0.19.2 (rust engine) and bumping up to v0.25.4.

@ion-elgreco is there any additional info I can provide to narrow things down?

Unknown error writing to Delta Lake: Generic S3 error: Error after 10 retries in 1.909096604s, max_retries:10, retry_timeout:180s, source:error sending request for url (https://s3.us-east-1.amazonaws.com/.../_delta_log/_last_checkpoint)

[ion-elgreco]

We're seeing similar errors on our S3 bucket on delta tables that were written on v0.19.2 (rust engine) and bumping up to v0.25.4.

@ion-elgreco is there any additional info I can provide to narrow things down?

Unknown error writing to Delta Lake: Generic S3 error: Error after 10 retries in 1.909096604s, max_retries:10, retry_timeout:180s, source:error sending request for url (https://s3.us-east-1.amazonaws.com/.../_delta_log/_last_checkpoint)

Rust log debugs would help

[liamphmurphy]

Nothing jumping out at me... I see the individual retry errs:

[2025-03-12T18:41:43Z INFO  object_store::client::retry] Encountered transport error backing off for 0.40535888 seconds, retry 10 of 10: error sending request for url (https://s3.us-east-1.amazonaws.com/.../_delta_log/_last_checkpoint)

The only thing that looks a little goofy to me is the S3 config indicates allow_unsafe_rename = true which I have confirmed we don't have that environment variable set, but maybe a red herring?

DEBUG deltalake_aws::storage] Initialized the object store: S3StorageBackend { allow_unsafe_rename: true, inner: AmazonS3...

[ion-elgreco]

@liamphmurphy How did you configure the store? S3StorageBackend is only created when you set the locking provider key or allow_unsafe_rename in the storage options

[ion-elgreco]

@liamphmurphy you can use this branch to see the full error message: #3328

[djouallah]

at least in Fabric,, it seems to be a certificate issue because of the use of Rustls, I had the same issue of obstore which uses object store, both cases, windows work fine

using "allow_invalid_certificates":"true" fixed the issue

[ion-elgreco]

@liamphmurphy did you have a chance to look at that branch to see the full error message

[liamphmurphy]

Hi @ion-elgreco , apologies for the delay. I seem to be having similar SSL issues, on your branch I get this error:

[ERROR] SSLError: SSL validation failed for https://<aws-s3-path-.../file.json

In terms of setup, this is deployed as an AWS Lambda. All we do is provide these env vars:

• AWS_S3_LOCKING_PROVIDER=dynamodb
• DELTA_DYNAMO_TABLE_NAME=delta-writer-lock-table

Is providing that locking provider an outdated method? This is how it is configured now and working in v0.19.2.

[ion-elgreco]

Yeah locking providers aren't needed anymore, all S3 supports putIfAbsent now including AWS.

Try this older branch perhaps: https://github.com/ion-elgreco/delta-rs/tree/fix/rustls-with-aws-lc-s