Delta version 3.3.0 having dependency issues with spark 3.5.x

[v0g03kp]

mvnDependencyTree.txt

fullLogs.txt

Bug

Which Delta project/connector is this regarding?

• Spark

Describe the problem

Using latest delta version 3.3.0 and spark (3.5.x), dependency issues are there.

25/01/20 10:32:50 ERROR Client: Application diagnostics message: User class threw exception: java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/plans/logical/SupportsNonDeterministicExpression
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:555)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:555)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at io.delta.sql.DeltaSparkSessionExtension.$anonfun$apply$9(DeltaSparkSessionExtension.scala:119)
at org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildPostHocResolutionRules$1(SparkSessionExtensions.scala:211)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)

Steps to reproduce

Set up an environment in scala(2.12.18) using delta 3.3.0 and spark 3.5.x

Observed results

A run time exception is thrown.
25/01/20 10:32:50 ERROR Client: Application diagnostics message: User class threw exception: java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/plans/logical/SupportsNonDeterministicExpression
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:555)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451

Expected results

It should not break , and data gets loaded to target destination.

Further details

Attached full error logs and maven dependency tree details.

Environment information

• Delta Lake version: 3.3.0
• Spark version: 3.5.0 / 3.5.1 / 3.5.2 / 3.5.3
• Scala version: 2.12.18

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

• Yes. I can contribute a fix for this bug independently.
• Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
• No. I cannot contribute a bug fix at this time.

[felipepessoto]

Although doc (https://docs.delta.io/latest/releases.html) says 3.3.0 is compatible with Spark 3.5.X, I'm not completely sure, Spark minor versions sometimes has binary incompatibilities. (https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html)

The class in the error, SupportsNonDeterministicExpression, was introduced in Spark 3.5.2. Your mvnDependencyTree.txt says you built Delta with Spark 3.5.2.

Is your runtime also running with Spark 3.5.2? If you use Spark <=3.5.1 to run it, I guess this error may happen