Skip to content
This repository was archived by the owner on Feb 27, 2025. It is now read-only.

append with bulk insert does results in NoSuchMethodError with DB runtime 12.2 LTS while overwrite works #263

Open
eschnepel opened this issue Jul 23, 2024 · 1 comment

Comments

@eschnepel
Copy link

I am inserting data into a Azure SQL DB table using bulk inserts with mode "append".

When the table does not exist before the table is created as expected.
When the table does already exist I am getting the following error:

An error occurred while calling o6686.save.
: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema$default$4()Z
at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)
at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)
at com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)
....

some searching around, I found the commit
apache/spark@7021806#diff-c3859e97335ead4b131263565c987d877bea0af3adbd6c5bf2d3716768d2e083
which changed the interface for the getSchema function. But the commit relates only to the master branch is tagged for release 4.0.0.

@ANSHUL22122001
Copy link

Did you get any solution for this?
we are also facing the same issue, when we upgraded pyspark version to 3.4

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants