Charmed Apache Spark is a solution that makes operating Apache Spark workloads on Kubernetes seamless, secure, and production-ready. This solution includes:
- charmed operators for several components in the Apache Spark ecosystems, bundled together in the Charmed Apache Spark bundle: Charmcraft | GitHub - this repository
- Client tools snap for Apache Spark: snapcraft | GitHub
- spark8t Python library
This repository contains relevant artifacts for deploying and testing Charmed Apache Spark:
- python — contains the
spark-testpackage that provides utilities and fixtures to easily implement Charmed Apache Spark tests. Find more information in its readme file - releases — contains the artifacts for the different channels, supporting the following backends:
- yaml — using Juju YAML bundles to easily deploy Charmed Apache Spark on K8s
- terraform — using Terraform scripts to deploy Charmed Apache Spark using the Juju Terraform provider
Charmed Apache Spark bundle is also available on Charmhub.
The minimum requirements are as follows:
- 8 GB of HDD.
- 2 CPU threads per host.
- Access to the internet for downloading the Canonical Apache Spark Image.
- Access to a Kubernetes cluster, e.g. MicroK8s or Charmed Kubernetes.
A production-ready solution might require more resources.
Charmed Apache Spark supports native integration with the Canonical Observability Stack (COS). If you want to enable monitoring on top of Charmed Apache Spark, make sure that you have a Juju model with COS correctly deployed and see the How to enable monitoring guide. To deploy COS on MicroK8s, follow the step-by-step tutorial. For more information about Charmed Apache Spark and COS integration, refer to the COS documentation and the monitoring explanation section.
For information on security features and the use of cryptography, see the Security explanation page.
Security issues in the Charmed Apache Spark can be reported through LaunchPad. Please do not file GitHub issues about security issues.
Canonical welcomes contributions to Charmed Apache Spark. Please check out our contribution guidelines if you're interested in contributing to the solution. If you truly enjoy working on open-source projects like this one and you would like to be part of the OSS revolution, please don't forget to check out the career opportunities we have at Canonical.
The Charmed Apache Spark is free software, distributed under the Apache Software License, version 2.0.
See LICENSE for more information.