feat!: update DataFusion to 45.0 and Arrow to 54.1 #3503

timsaucer · 2025-03-03T11:46:48Z

This PR updates DataFusion to 45.0 and Arrow to 54.1.

The update to Arrow required updating PyO3 for the python package. This had a series of breaking changes to their API.

github-actions · 2025-03-03T11:47:07Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

codecov-commenter · 2025-03-03T20:22:37Z

Codecov Report

Attention: Patch coverage is 87.09677% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.49%. Comparing base (9888678) to head (9a0ef7c).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/write.rs	70.00%	3 Missing ⚠️
rust/lance/src/dataset.rs	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3503      +/-   ##
==========================================
- Coverage   78.49%   78.49%   -0.01%     
==========================================
  Files         253      253              
  Lines       94542    94543       +1     
  Branches    94542    94543       +1     
==========================================
- Hits        74213    74211       -2     
- Misses      17319    17327       +8     
+ Partials     3010     3005       -5

Flag	Coverage Δ
unittests	`78.49% <87.09%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eddyxu · 2025-03-05T15:11:18Z

We need to make this as a breaking change.

timsaucer · 2025-03-05T16:25:13Z

@eddyxu I tried adding a breaking change text in a commit to see if the github actions bot would add the tag, but that didn't seem to make an impact. Can you add the tag or do I need to put something in the description?

timsaucer · 2025-03-05T19:13:37Z

Looks like CI kept failing because there were changes on main that I didn't have yet, so rebased and corrected them.

…x<>> in a few places and shifting over to owned data in a couple of structs

… impacts primarily the python crate

westonpace

Thank you so much for taking this on. I had made an attempt earlier and gotten daunted by all the python changes.

I have a few minor suggestions but overall things look good

python/src/dataset.rs

python/src/tracing.rs

python/src/file.rs

python/src/transaction.rs

westonpace · 2025-03-06T17:32:50Z

python/src/transaction.rs

+        let old_id: String = ob.getattr("old_id")?.extract()?;
+        let new_id: String = ob.getattr("new_id")?.extract()?;


Why is it that extract works here I wonder? I feel like all the other string extract were changed to downcast

python/src/tracing.rs

wjones127

Thanks for working on this. I recognize the PyO3 changes were a lot.

I would like to not have to modify WriteDestination like that. It makes the changes rather large. I found an alternative solution using pyo3's PyBackedStr, and have put up a commit here: rerun-io@aa14dd3

Cargo.toml

python/src/file.rs

rust/lance/src/dataset/write.rs

timsaucer · 2025-03-06T21:53:37Z

Thank you both for the reviews, and especially for that commit. That is a much nicer solution than I came up with. Can one of you run the workflow again?

westonpace

One last tiny nit and then I'm good. Thanks again for taking this task on :)

.github/workflows/rust.yml

Co-authored-by: Weston Pace <[email protected]>

westonpace · 2025-03-07T00:35:42Z

The linux-arm failure is a timeout. We can ignore for now.

wjones127

Thanks for following up on those changes. Great work here.

westonpace · 2025-03-08T02:23:15Z

Sorry, I just noticed the PR title was feat!. Did this PR actually contain any breaking changes? I don't think it did.

eddyxu · 2025-03-08T02:46:07Z

Bumping arrow counts as breaking change iiuc?

timsaucer · 2025-03-10T14:54:26Z

Also, anyone who uses the python crate as a dependency will be forced to update pyo3 which introduces many deprecations.

westonpace · 2025-03-10T18:37:23Z

Bumping arrow counts as breaking change iiuc?

Also, anyone who uses the python crate as a dependency will be forced to update pyo3 which introduces many deprecations.

Fair points. Some of our APIs take arrow data directly so our users would need to stay in lock-step.

timsaucer changed the title ~~Bump DataFusion to 45.0~~ feat: Update DataFusion to 45.0 Mar 3, 2025

github-actions bot added the enhancement label Mar 3, 2025

timsaucer mentioned this pull request Mar 3, 2025

Rerun PRs rerun-io/opensource#2

Open

timsaucer marked this pull request as draft March 4, 2025 17:59

github-actions bot added the python label Mar 5, 2025

timsaucer changed the title ~~feat: Update DataFusion to 45.0~~ feat: update DataFusion to 45.0 and Arrow to 54.1 Mar 5, 2025

eddyxu changed the title ~~feat: update DataFusion to 45.0 and Arrow to 54.1~~ feat!: update DataFusion to 45.0 and Arrow to 54.1 Mar 5, 2025

eddyxu requested review from westonpace, wjones127 and BubbleCal March 5, 2025 16:39

github-actions bot added the breaking-change label Mar 5, 2025

timsaucer force-pushed the feat/datafusion-45 branch from 8136543 to 4923988 Compare March 5, 2025 19:09

timsaucer marked this pull request as ready for review March 5, 2025 20:28

timsaucer added 12 commits March 6, 2025 12:41

Bump DataFusion to 45.0

5a7ae59

Cargo fmt

b676bc0

Update dependencies for arrow 54

77841d0

Python requires all pyclass objects to be Sync, so adding in Arc<Mute…

b0c5413

…x<>> in a few places and shifting over to owned data in a couple of structs

Resolving warnings due to pyo3 update and deprecated features

ab31e9d

Updating pyo3 deprecated many functions

78395f1

Resolving test issues, mostly around string changes

c7d9773

Cargo fmt

07b801c

Update MSRV in accordance with datafusion

e547abc

Resolve clippy warning

476f203

Clippy warning

c72f21c

BREAKING CHANGE this PR updates PyO3 which has breaking changes which…

862fae9

… impacts primarily the python crate

timsaucer added 2 commits March 6, 2025 12:41

Updated clippy added a new check

9a9cfe6

Fix clippy warning after rebase

Loading
Loading status checks…

75f4222

timsaucer force-pushed the feat/datafusion-45 branch from 9a0ef7c to 75f4222 Compare March 6, 2025 17:41

westonpace requested changes Mar 6, 2025

View reviewed changes

wjones127 mentioned this pull request Mar 6, 2025

Simplification rerun-io/lance#3

Closed

wjones127 requested changes Mar 6, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

python/src/file.rs Outdated Show resolved Hide resolved

rust/lance/src/dataset/write.rs Outdated Show resolved Hide resolved

wjones127 and others added 5 commits March 6, 2025 16:18

revert irrelevant changes

107aa4e

impl Sync not required

a2ed1cd

Revert msrv

Loading
Loading status checks…

f7a1be5

More moving to PyBackedStr

Loading
Loading status checks…

f62b580

Remove &mut that is no longer necessary due to the mutex

Loading
Loading status checks…

524a252

Revert clippy change for mrsv 1.82

Loading
Loading status checks…

ec7a8ea

westonpace approved these changes Mar 6, 2025

View reviewed changes

.github/workflows/rust.yml Outdated Show resolved Hide resolved

Update .github/workflows/rust.yml

Loading
Loading status checks…

b4c7f51

Co-authored-by: Weston Pace <[email protected]>

wjones127 approved these changes Mar 7, 2025

View reviewed changes

wjones127 merged commit 3b9d546 into lancedb:main Mar 7, 2025
26 of 27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: update DataFusion to 45.0 and Arrow to 54.1 #3503

feat!: update DataFusion to 45.0 and Arrow to 54.1 #3503

timsaucer commented Mar 3, 2025 •

edited

Loading

github-actions bot commented Mar 3, 2025

codecov-commenter commented Mar 3, 2025 •

edited

Loading

eddyxu commented Mar 5, 2025

timsaucer commented Mar 5, 2025

timsaucer commented Mar 5, 2025

westonpace left a comment

westonpace Mar 6, 2025

wjones127 left a comment

timsaucer commented Mar 6, 2025

westonpace left a comment

westonpace commented Mar 7, 2025

wjones127 left a comment

westonpace commented Mar 8, 2025

eddyxu commented Mar 8, 2025

timsaucer commented Mar 10, 2025

westonpace commented Mar 10, 2025

		let old_id: String = ob.getattr("old_id")?.extract()?;
		let new_id: String = ob.getattr("new_id")?.extract()?;

feat!: update DataFusion to 45.0 and Arrow to 54.1 #3503

feat!: update DataFusion to 45.0 and Arrow to 54.1 #3503

Conversation

timsaucer commented Mar 3, 2025 • edited Loading

github-actions bot commented Mar 3, 2025

codecov-commenter commented Mar 3, 2025 • edited Loading

Codecov Report

eddyxu commented Mar 5, 2025

timsaucer commented Mar 5, 2025

timsaucer commented Mar 5, 2025

westonpace left a comment

Choose a reason for hiding this comment

westonpace Mar 6, 2025

Choose a reason for hiding this comment

wjones127 left a comment

Choose a reason for hiding this comment

timsaucer commented Mar 6, 2025

westonpace left a comment

Choose a reason for hiding this comment

westonpace commented Mar 7, 2025

wjones127 left a comment

Choose a reason for hiding this comment

westonpace commented Mar 8, 2025

eddyxu commented Mar 8, 2025

timsaucer commented Mar 10, 2025

westonpace commented Mar 10, 2025

timsaucer commented Mar 3, 2025 •

edited

Loading

codecov-commenter commented Mar 3, 2025 •

edited

Loading