Filter rows directly from pa.RecordBatch

[gabeiglio]

This PR from Apache Arrow was merged to allow to filter with a boolean expression directly on pa.RecordBatch.

I believe pyiceberg is currently using pyarrow version 19.0.0.
Filtering from pa.RecordBatch was introduced in python in version 17.0.0

I have not run integration tests for some reason my docker setup is messed up. I believe this test should check this change:

iceberg-python/tests/integration/test_deletes.py

Line 314 in dfbee4b

def test_read_multiple_batches_in_task_with_position_deletes(spark: SparkSession, session_catalog: RestCatalog) -> None:

Closes #1050

[Fokko]

Thanks for fixing this @gabeiglio

In addition, I think we also need to bump the minimal version of Arrow here:

iceberg-python/pyproject.toml

Line 64 in dfbee4b

pyarrow = { version = ">=14.0.0,<20.0.0", optional = true }

[kevinjqliu]

thanks for following up on that comment 😄

if we're bumping minimum pyarrow version to 17, we might want to address this comment as well
https://github.com/apache/iceberg-python/pull/1621/files#diff-8d5e63f2a87ead8cebe2fd8ac5dcf2198d229f01e16bb9e06e21f7277c328abdR1335-R1338

[gabeiglio]

@kevinjqliu IIUC removing the schema casting will allow pyarrow scanner to infer by itself if it needs or not large types? So it is basically a matter of changing the assertions in tests to the types of the result of the scan?

[kevinjqliu]

I believe so. We can also do this in a follow up PR! I just saw that comment during code review

[kevinjqliu]

Looks like theres an issue in CI tests

[gabeiglio]

Yes, I think it would be better to split these changes in separate PRs since there are a lot of changes to be made to tests specially. (If thats okay ill open the other PR for schema casting @kevinjqliu @Fokko)

[Fokko]

One minor comment, looks great, and so much cleaner :)

[Fokko]

nit, I think we can drop the batch:

Suggested change

	for batch in batches:
	for current_batch in batches:

[kevinjqliu]

looks like CIs broken on poetry

[kevinjqliu]

is this logically equivalent? feels like there was a reason to write it the other way.

cc @sungwy do you have context on this?

[gabeiglio]

Oh I wasn't planning on pushing this change 🤦. I'll revert it in the next commit if we want