Dagster Cloud Hybrid Deployment Quickstart

This template lets you get started using Dagster Cloud with a Hybrid agent.

Note It is recommended to first deploy the example project included in this repository and then replace it with your own Dagster project.

Pre-requisites

What you need to start using this template:

A Dagster Cloud account set up for Hybrid deployments.
A Hybrid agent up and running.
A Docker container registry accessible from the hybrid agent and from your GitHub workflows.

Step 1. Create a new repository from this template

Click the Use this Template button and provide details for your new repo.

Step 2. Add your Docker registry to `dagster_cloud.yaml`

The dagster_cloud.yaml file defines the configuration for building and deploying your code locations. For the quickstart_etl, specify the Docker registry in the registry: key:

dagster-cloud-hybrid-quickstart/dagster_cloud.yaml

Line 7 in 38c16dd

registry: <account-id>.dkr.ecr.us-west-2.amazonaws.com/<image-name>

Step 3. Modify the GitHub Workflow

Edit the GitHub Workflow at .github/workflows/dagster-cloud-deploy.yml to configure your Dagster Cloud account as well as Docker registry access.

Set the DAGSTER_CLOUD_ORGANIZATION environment to the name of your Dagster Cloud organization. If you access Dagster Cloud at https://acme.dagster.cloud then your organization is acme.

dagster-cloud-hybrid-quickstart/.github/workflows/dagster-cloud-deploy.yml

Lines 15 to 16 in 38c16dd

# The organization name in Dagster Cloud

DAGSTER_CLOUD_ORGANIZATION: "<organization-name>"

Set the IMAGE_REGISTRY environment to the same registry specified in dagster_cloud.yaml:

dagster-cloud-hybrid-quickstart/.github/workflows/dagster-cloud-deploy.yml

Lines 23 to 24 in 38c16dd

    
           # The IMAGE_REGISTRY should match the 'registry:'' in dagster_cloud.yaml 
        
           IMAGE_REGISTRY: "<account-id>.dkr.ecr.us-west-2.amazonaws.com/dagster-cloud-image"

Uncomment one of the options for logging into the Docker registry:

dagster-cloud-hybrid-quickstart/.github/workflows/dagster-cloud-deploy.yml

Lines 70 to 114 in 38c16dd

    
           # Building and deploying the docker image requires a login step specific to the container 
        
           # registry. 
        
           # Multiple examples are provided below. 
        
           # # AWS ECR 
        
           # # https://github.com/aws-actions/amazon-ecr-login 
        
           # - name: Configure AWS credentials 
        
           #   if: steps.prerun.outputs.result != 'skip' 
        
           #   uses: aws-actions/configure-aws-credentials@v2 
        
           #   with: 
        
           #     aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} 
        
           #     aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} 
        
           #     aws-region:  ${{ secrets.AWS_REGION }} 
        
           # - name: Login to ECR 
        
           #   if: steps.prerun.outputs.result != 'skip' 
        
           #   uses: aws-actions/amazon-ecr-login@v1 
        
           # # DockerHub 
        
           # # https://github.com/docker/login-action#docker-hub 
        
           # - name: Login to Docker Hub 
        
           #   if: steps.prerun.outputs.result != 'skip' 
        
           #   uses: docker/login-action@v1 
        
           #   with: 
        
           #     username: ${{ secrets.DOCKERHUB_USERNAME }} 
        
           #     password: ${{ secrets.DOCKERHUB_TOKEN }} 
        
           # # GitHub Container Registry 
        
           # # https://github.com/docker/login-action#github-container-registry 
        
           # - name: Login to GitHub Container Registry 
        
           #   if: steps.prerun.outputs.result != 'skip' 
        
           #   uses: docker/login-action@v1 
        
           #   with: 
        
           #     registry: ghcr.io 
        
           #     username: ${{ github.actor }} 
        
           #     password: ${{ secrets.GITHUB_TOKEN }} 
        
           # # GCR 
        
           # # https://github.com/docker/login-action#google-container-registry-gcr 
        
           # - name: Login to GCR 
        
           #   if: steps.prerun.outputs.result != 'skip' 
        
           #   uses: docker/login-action@v1 
        
           #   with: 
        
           #     registry: gcr.io 
        
           #     username: _json_key 
        
           #     password: ${{ secrets.GCR_JSON_KEY }}

Step 4. Set up secrets

Set up secrets on your newly created repository by navigating to the Settings panel in your repo, clicking Secrets on the sidebar, and selecting Actions. Then, click New repository secret. The following secrets are needed.

Name	Description
`DAGSTER_CLOUD_API_TOKEN`	An agent token, for more details see the Dagster Cloud docs.
Docker access secrets	Depending on which Docker registry you are using, you must define the credentials listed in the workflow file.

Here is an example screenshot showing the secrets for AWS ECR.

Step 5. Verify builds are successful

At this point, the workflow run should complete successfully and you should see the quickstart_etl location in https://dagster.cloud. If builds are failing, ensure that your secrets are properly set up.

Add or modify code locations

Once you have the quickstart_etl example deployed, you can replace the sample code with your Dagster project. You will then need to update the dagster_cloud.yaml file:

Update dagster_cloud.yaml. See documentation for details.
If you have more than one code location, duplicate the build-docker-image and the "ci set-build-output" steps in dagster-cloud-deploy.yaml for the new code locations.

Advanced customization

Disable branch deployments

Branch Deployments are enabled by default. To disable them comment out the for your Hybrid agent, comment out the pull_request section in dagster_cloud.yaml:

dagster-cloud-hybrid-quickstart/.github/workflows/dagster-cloud-deploy.yml

Lines 7 to 8 in 9f63f62

    
           pull_request:  # For branch deployments 
        
              types: [opened, synchronize, reopened, closed]

Customize the Docker build process

A standard Dockerfile is included in this project and used to build the quickstart_etl. This file is used by the build-push-action:

dagster-cloud-hybrid-quickstart/.github/workflows/dagster-cloud-deploy.yml

Lines 123 to 129 in fa0a0d3

    
           - name: Build and upload Docker image for "example_location" 
        
             if: steps.prerun.outputs.result != 'skip' 
        
             uses: docker/build-push-action@v4 
        
             with: 
        
               context: . 
        
               push: true 
        
               tags: ${{ env.IMAGE_REGISTRY }}:${{ env.IMAGE_TAG }}-example-location

To customize the Docker image, modify the build-push-action and update the Dockerfile as needed:

To use a different directory for the Dockerfile, use the context: input. See build-push-action for more details.
To reuse a Docker image for multiple code locations, use a single build-push-action and multiple "ci set-build-output" steps, all using the same image tag.

Deploy a subset of code locations

The ci-init step accepts a location_names input string containing a JSON list of location names to be deployed. To deploy only specific locations provide the location_names: input, for example:

      - name: Initialize build session
        id: ci-init
        if: steps.prerun.outputs.result != 'skip'
        uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
        with:
          project_dir: ${{ env.DAGSTER_PROJECT_DIR }}
          dagster_cloud_yaml_path: ${{ env.DAGSTER_CLOUD_YAML_PATH }}
          deployment: 'prod'
          location_names: '["quickstart_etl1", "location2"]'  # only deploy these two locations

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
.github/workflows		.github/workflows
quickstart_etl		quickstart_etl
quickstart_etl_tests		quickstart_etl_tests
Dockerfile		Dockerfile
README.md		README.md
dagster_cloud.yaml		dagster_cloud.yaml
pyproject.toml		pyproject.toml
setup.cfg		setup.cfg
setup.py		setup.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Dagster Cloud Hybrid Deployment Quickstart

Pre-requisites

Step 1. Create a new repository from this template

Step 2. Add your Docker registry to `dagster_cloud.yaml`

Step 3. Modify the GitHub Workflow

Step 4. Set up secrets

Step 5. Verify builds are successful

Add or modify code locations

Advanced customization

Disable branch deployments

Customize the Docker build process

Deploy a subset of code locations

About

Releases

Packages

Contributors 4

Languages

	# The organization name in Dagster Cloud
	DAGSTER_CLOUD_ORGANIZATION: "<organization-name>"

	# The IMAGE_REGISTRY should match the 'registry:'' in dagster_cloud.yaml
	IMAGE_REGISTRY: "<account-id>.dkr.ecr.us-west-2.amazonaws.com/dagster-cloud-image"

	# Building and deploying the docker image requires a login step specific to the container
	# registry.
	# Multiple examples are provided below.

	# # AWS ECR
	# # https://github.com/aws-actions/amazon-ecr-login
	# - name: Configure AWS credentials
	# if: steps.prerun.outputs.result != 'skip'
	# uses: aws-actions/configure-aws-credentials@v2
	# with:
	# aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
	# aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	# aws-region: ${{ secrets.AWS_REGION }}
	# - name: Login to ECR
	# if: steps.prerun.outputs.result != 'skip'
	# uses: aws-actions/amazon-ecr-login@v1

	# # DockerHub
	# # https://github.com/docker/login-action#docker-hub
	# - name: Login to Docker Hub
	# if: steps.prerun.outputs.result != 'skip'
	# uses: docker/login-action@v1
	# with:
	# username: ${{ secrets.DOCKERHUB_USERNAME }}
	# password: ${{ secrets.DOCKERHUB_TOKEN }}

	# # GitHub Container Registry
	# # https://github.com/docker/login-action#github-container-registry
	# - name: Login to GitHub Container Registry
	# if: steps.prerun.outputs.result != 'skip'
	# uses: docker/login-action@v1
	# with:
	# registry: ghcr.io
	# username: ${{ github.actor }}
	# password: ${{ secrets.GITHUB_TOKEN }}

	# # GCR
	# # https://github.com/docker/login-action#google-container-registry-gcr
	# - name: Login to GCR
	# if: steps.prerun.outputs.result != 'skip'
	# uses: docker/login-action@v1
	# with:
	# registry: gcr.io
	# username: _json_key
	# password: ${{ secrets.GCR_JSON_KEY }}

	pull_request: # For branch deployments
	types: [opened, synchronize, reopened, closed]

	- name: Build and upload Docker image for "example_location"
	if: steps.prerun.outputs.result != 'skip'
	uses: docker/build-push-action@v4
	with:
	context: .
	push: true
	tags: ${{ env.IMAGE_REGISTRY }}:${{ env.IMAGE_TAG }}-example-location

dagster-io/dagster-cloud-hybrid-quickstart

Folders and files

Latest commit

History

Repository files navigation

Dagster Cloud Hybrid Deployment Quickstart

Pre-requisites

Step 1. Create a new repository from this template

Step 2. Add your Docker registry to dagster_cloud.yaml

Step 3. Modify the GitHub Workflow

Step 4. Set up secrets

Step 5. Verify builds are successful

Add or modify code locations

Advanced customization

Disable branch deployments

Customize the Docker build process

Deploy a subset of code locations

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 4

Languages

Step 2. Add your Docker registry to `dagster_cloud.yaml`

Packages