-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Coral-Trino: Migrate function operator transformers defined in CalciteTrinoUDFMap from RelNode layer to SqlNode layer #349
Conversation
…elNode layer to SqlNode layer
coral-common/src/main/java/com/linkedin/coral/common/transformers/SqlCallTransformers.java
Outdated
Show resolved
Hide resolved
...mon/src/main/java/com/linkedin/coral/common/transformers/StandardUDFOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...mon/src/main/java/com/linkedin/coral/common/transformers/StandardUDFOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/TrinoSqlUDFConverter.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/TrinoSqlUDFConverter.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
The class |
I think this PR migrates operators beyond ones leveraging JSON infra right? Might consider updating the title/description if necessary. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @yiqiangin for this PR!
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/Calcite2TrinoUDFConverter.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
.../java/com/linkedin/coral/trino/rel2trino/transfomers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
.../java/com/linkedin/coral/trino/rel2trino/transfomers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
.../java/com/linkedin/coral/trino/rel2trino/transfomers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
.../java/com/linkedin/coral/trino/rel2trino/transfomers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
Ideally, everything is just a |
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CalciteTrinoUDFOperatorTransformers.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @yiqiangin, please run regression test for Trino again after addressing the comments.
coral-common/src/main/java/com/linkedin/coral/common/transformers/SqlCallTransformers.java
Outdated
Show resolved
Hide resolved
coral-common/src/main/java/com/linkedin/coral/common/transformers/SqlCallTransformers.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/TrinoSqlUDFConverter.java
Outdated
Show resolved
Hide resolved
.../java/com/linkedin/coral/trino/rel2trino/transfomers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/linkedin/coral/trino/rel2trino/transfomers/ToDateOperatorTransformer.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/linkedin/coral/trino/rel2trino/transfomers/ToDateOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...n/java/com/linkedin/coral/common/transformers/SignatureBasedConditionSqlCallTransformer.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
...no/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
@yiqiangin: please resolve the conflicts. |
What is the objective of |
.../main/java/com/linkedin/coral/trino/rel2trino/utils/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
...rino/src/main/java/com/linkedin/coral/trino/rel2trino/utils/TrinoSqlCallTransformerUtil.java
Outdated
Show resolved
Hide resolved
...rino/src/main/java/com/linkedin/coral/trino/rel2trino/utils/TrinoSqlCallTransformerUtil.java
Outdated
Show resolved
Hide resolved
...n/java/com/linkedin/coral/common/transformers/SignatureBasedConditionSqlCallTransformer.java
Outdated
Show resolved
Hide resolved
@wmoustafa From my understanding, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @yiqiangin, please rerun the regression test after addressing all the comments.
coral-common/src/main/java/com/linkedin/coral/common/transformers/SqlCallTransformer.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/linkedin/coral/trino/rel2trino/transfomers/ToDateOperatorTransformer.java
Outdated
Show resolved
Hide resolved
.../main/java/com/linkedin/coral/trino/rel2trino/utils/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
.../main/java/com/linkedin/coral/trino/rel2trino/utils/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
.../main/java/com/linkedin/coral/trino/rel2trino/utils/CoralToTrinoSqlCallTransformersUtil.java
Outdated
Show resolved
Hide resolved
.../java/com/linkedin/coral/trino/rel2trino/transfomers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @yiqiangin for working on this PR!
...src/main/java/com/linkedin/coral/trino/rel2trino/transformers/ToDateOperatorTransformer.java
Show resolved
Hide resolved
...java/com/linkedin/coral/trino/rel2trino/transformers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...java/com/linkedin/coral/trino/rel2trino/transformers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...java/com/linkedin/coral/trino/rel2trino/transformers/MapStructAccessOperatorTransformer.java
Outdated
Show resolved
Hide resolved
...com/linkedin/coral/trino/rel2trino/transformers/LinkedInOperatorBasedSqlCallTransformer.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallConverter.java
Outdated
Show resolved
Hide resolved
...on/src/main/java/com/linkedin/coral/common/transformers/OperatorBasedSqlCallTransformer.java
Outdated
Show resolved
Hide resolved
coral-common/src/main/java/com/linkedin/coral/common/calcite/CalciteUtil.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallConverter.java
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallConverter.java
Outdated
Show resolved
Hide resolved
...com/linkedin/coral/trino/rel2trino/transformers/LinkedInOperatorBasedSqlCallTransformer.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallConverter.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallConverter.java
Outdated
Show resolved
Hide resolved
...on/src/main/java/com/linkedin/coral/common/transformers/OperatorBasedSqlCallTransformer.java
Outdated
Show resolved
Hide resolved
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/CoralToTrinoSqlCallConverter.java
Outdated
Show resolved
Hide resolved
I have simplified the APIs. You can take a look here. |
coral-common/src/main/java/com/linkedin/coral/common/transformers/SqlIdentifierTransformer.java
Outdated
Show resolved
Hide resolved
...n/src/main/java/com/linkedin/coral/common/transformers/OperatorRenameSqlCallTransformer.java
Show resolved
Hide resolved
...coral/trino/rel2trino/transformers/GenericCoralRegistryOperatorRenameSqlCallTransformer.java
Show resolved
Hide resolved
This reverts commit 440aa1c.
coral-trino/src/main/java/com/linkedin/coral/trino/rel2trino/Calcite2TrinoUDFConverter.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have other major concerns. Let's merge if no major concerns from @wmoustafa.
LGTM. Great work @yiqiangin! Thanks! |
The code changes include:
OperatorBasedSqlCallTransformer
which extendsSqlCallTransformer
and its condition is implemented based on the input operator name and the number of the operandsLinkedInOperatorBasedSqlCallTransformer
which is a subclass of OperatorBasedSqlCallTransformer and transforms a Coral operator into a Trino operator on SqlNode layer for the LinkedIn specific functionsMapStructAccessOperatorTransformer
which is an ad-hoc SqlCallTransformer which converts the map struct access operator "[]" defined from Calcite in a SqlIdentifier into a UDF operator of "element_at", e.g. from "col"["field"] to element_at("col", "field")CalciteTrinoUDFMap
in the class ofCalcite2TrinoUDFConverter
on RelNode layerCoralToTrinoSqlCallConverter
which extends the class ofSqlShuttle
and initialize a class ofSqlCallTransformers
which containing a list ofSqlCallTransformer
to traverse the hierarchy of a SqlCall and converts the functions from Coral operator to Trino operator if it is requiredCoralToTrinoSqlCallConverter
to convert function operators on SqlNode layer in the class ofRelToTrinoConverter
ToDateOperatorTransformer
as an Ad Hoc SqlCallTransformer as it needs to check the configuration parameter ofAVOID_TRANSFORM_TO_DATE_UDF
in its condition functionTo be noted that the migration from UDF operator transformers based on JSON infra to those implemented in Java code will be done in another several separated PRs