Implement authorization in DataFusion

[shohamyamin]

Is your feature request related to a problem or challenge?

The current problem is that I need to have the ability to do row level security and column level security and column mask and currently I can’t do that.

Trino allows that using their opa plugin and authorization in the Trino engine.

Describe the solution you'd like

Please support and allow making authorization in datafusion making it a candidate to replace Trino in cases where rls authorization logic is needed.

So every query sent to DataFusion will be parsed and raise an authorization process checking user access and applying rls and column mask if needed.

Describe alternatives you've considered

Use Trino as a query engine

Additional context

The main usecase is authorization over iceberg based datalake

[jayzhan211]

Isn't authorization Catalog's role? What would it be like to have authorization logic in query engine? 🤔

[shohamyamin]

In catalog you can have only table level authorization(you have access to a table or not) but you cant have row level as it required to do some compute on the data(filter it for row level or masking the data for column mask .etc)

the best case and there where things are going the catalog will return the authorization logic and the compute engine(need to be a trusted compute engine) will do the authorization logic based on the returned logic from the catalog.

But still the engine will need to have a way to do the authorization logic.

Here is a link to a proposal for Support for Row Filtering and Column Masking in REST Specification so the catalog will returned the row level and the column masking expression:
https://docs.google.com/document/d/1NHr32sMwr8TH_dUJjQLcbEv8bAVbuJCn5pHqe1I_xIE/edit?tab=t.0