Added support for Polars DataFrame and LazyFrame

[yigal-rozenberg]

Polars (https://pola.rs) is an open-source library for data manipulation, known for being one of the fastest data processing solutions on a single machine. It features a well-structured, typed API that is both expressive and easy to use.

this chnage is a simple 'to_polars' addiotn to the table api.

iceberg_table = catalog.load_table('data.data_points')
pdf = iceberg_table.scan().to_polars()
print(pdf)

[amitgilad3]

Really nice - i really like working with polars

[Fokko]

First of all, thanks @yigal-rozenberg for working on this. I'm open to adding this. Could you add a section on Polars to the docs as well?

See https://py.iceberg.apache.org/api/#query-the-data for examples. You can find the file here: https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/api.md

[yigal-rozenberg]

I will provide with the relevant documentation.

[corleyma]

At first glance, this approach seems fairly inferior to using the Polars scan_iceberg functionality, which:

• returns a LazyFrame
• on evaluation of the LazyFrame, will push down filters to pyiceberg to limit the data returned.

I think it might be better to document the existing polars functionality vs adding and documenting this pattern.

[yigal-rozenberg]

Thanks for the comment!
I can add this to the documentation as an alternative.
The motivations was to extend existing PyIceberg table API to align with 'to_pandas' as an example.
Polars 'scan_iceberg' uses PyIceberg to create the LazyFrame:
https://github.com/pola-rs/polars/blob/9359ed576d972dce257346fcd62c8857f3d23277/py-polars/polars/io/iceberg.py#L139
The filtering can be done in PyIceberg, so aren't the2 approaches similar?

[corleyma]

Polars 'scan_iceberg' uses PyIceberg to create the LazyFrame:
https://github.com/pola-rs/polars/blob/9359ed576d972dce257346fcd62c8857f3d23277/py-polars/polars/io/iceberg.py#L139
The filtering can be done in PyIceberg, so aren't the2 approaches similar?

The difference is the approach as documented is encouraging folks to write their own filter predicates for pyiceberg before materializing a dataframe with polars, whereas the "polars way" (as a lazy dataframe API) would be to just create the lazyframe, construct your compute graph with whatever polars predicates/etc make sense for you, and rely on polars to push that down at .collect() time to appropriately filter data before load where possible.

[corleyma]

Separately, rather than adding more library-specific conversion code, it might make sense for pyiceberg to start leveraging the PyCapsule protocol to allow any third party library (dataframe or otherwise) that supports Arrow data to seamlessly consume pyiceberg constructs.

Polars already supports the PyCapsule interface. See https://docs.pola.rs/user-guide/misc/arrow/#using-the-arrow-pycapsule-interface for details.

Implementing the interface on e.g. pyiceberg tables would allow them to be passed directly to dataframe init in polars, just like you can do a pyarrow table today. It also doesn't assume anything about polars support/doesn't add a dependency on polars.

[yigal-rozenberg]

love the idea!
Will research in how to implement.
In the mean time, I believe this specific change request is straight forward, and allow both DataFrame, and LazyFrames to be utilized by Polars.

[kevinjqliu]

Thanks for adding this! I've added a few comments

[kevinjqliu]

can you rebase off main? looks like theres a conflict

[kevinjqliu]

theres still conflict with main. could you also remove .vscode/settings.json?

[kevinjqliu]

LGTM. I fixed the merge conflict and some linter issues.

I'll let Fokko chime in before proceeding