feat: add support for SparkConnect #539
Open
+3,137
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
26 tasks
razvan
moved this from Development: In Progress
to Development: Waiting for Review
in Stackable Engineering
maltesander
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
maltesander
requested changes
Apr 9, 2025
| pub const CONNECT_GRPC_PORT: i32 = 15002; | ||
| pub const CONNECT_UI_PORT: i32 = 4040; | ||
|
|
||
| pub const DUMMY_SPARK_CONNECT_GROUP_NAME: &str = "default"; |
There was a problem hiding this comment.
This is defined in common.rs as well.
Comment on lines
113
to
104
| scs: &v1alpha1::SparkConnectServer, | ||
| config: &v1alpha1::ExecutorConfig, | ||
| pi: &ResolvedProductImage, | ||
| config_map: &ConfigMap, |
There was a problem hiding this comment.
Not a fan of the abbreviations. I can live with scs but pi is probably not how you want to refer an image :) ?
Comment on lines
404
to
415
| ServicePort { | ||
| name: Some(String::from("grpc")), | ||
| port: CONNECT_GRPC_PORT, | ||
| ..ServicePort::default() | ||
| }, | ||
| ServicePort { | ||
| name: Some(String::from("http")), | ||
| port: CONNECT_UI_PORT, | ||
| ..ServicePort::default() | ||
| }, | ||
| ]), |
There was a problem hiding this comment.
grpc and http are used as well in build_deployment as well, constant?
Comment on lines
+67
to
+77
| # TODO: cannot use the PySpark job from the ConfigMap because it breaks | ||
| # with a "iceberg.SparkWrite$WriterFactory" ClassNotfoundException. | ||
| # Use the app bundled within spark-connect-client instead | ||
| # "/app/simple-connect-app.py", | ||
| # | ||
| command: | ||
| [ | ||
| "/usr/bin/python", | ||
| "/stackable/spark-connect-examples/python/simple-connect-app.py", | ||
| "sc://spark-connect-server", | ||
| ] |
There was a problem hiding this comment.
What needs to be done to fix this?
Co-authored-by: Malte Sander <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Part of: #284
See issue above for implemented features.
Definition of Done Checklist
Author
Reviewer
Acceptance