You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am interested in understanding how I might be able to accomplish the same sort of thing in Dagster: using the Dagster CLI on a local developer machine to interact with a Dagster instance running in Kubernetes.
Some additional context, if relevant:
We have Dagster deployed in several separate Kubernetes clusters. Each cluster corresponds to an application environment and has an associated kubectl context (kubectl --context=dev / kubectl --context=qa and so on).
We have several distinct Dagster user code repositories, deployed using the dagster/dagster-user-repositories Helm chart. (We will have five or six user code repositories deployed this way in total, once we're fully migrated)
I expect that the Dagster UI is going to service a majority of our existing argo CLI use cases, once we migrate. But I want to be sure that I'm accounting for all of our existing processes as we do this migration.
What I've found so far
Some dagster CLI commands support --grpc-host and --grpc-port CLI options. These commands can be run against a port-forward against the Kubernetes user code deployment (not the Dagster instance / webserver). For example, for the dagster job list command, this works:
$ uvx dagster job list --grpc-host localhost --grpc-port 3030
Repository __repository__
*************************
Job: __ASSET_JOB
Ops: (Execution Order)
hello
world
But the CLI docs really don't include much detail on the supported options, or whether they need to be run against a user deployment or the instance/webserver. Since dagster job list can be run against a user repository port-forward, I expected that dagster asset list could be as well, but it can't - dagster asset list does not support --grpc-host / --grpc-port CLI options. This isn't intuitive to me - I would expect that if the job command can be run this way, that the asset command could be as well.
Maybe a better way to do this is via the dagster-graphql CLI command? That supports a --remote CLI option, can I use that with --remote localhost:8080 (if I have the a port-forward open to the dagster-webserver at 8080) ?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
(I had asked this in the Dagster Slack but thought it might be better asked here).
Context
We're in the process of migrating our data workloads from Argo Workflows to Dagster (Dagster OSS, with the
dagster/dagster
Helm chart).We have a lot of existing conventions and operational processes etc that use the
argo
CLI to run workloads.For example, something like "run a scheduled job right now"
or "submit a workflow from a template definition, with parameters"
My Goal
I am interested in understanding how I might be able to accomplish the same sort of thing in Dagster: using the Dagster CLI on a local developer machine to interact with a Dagster instance running in Kubernetes.
Some additional context, if relevant:
kubectl --context=dev
/kubectl --context=qa
and so on).dagster/dagster-user-repositories
Helm chart. (We will have five or six user code repositories deployed this way in total, once we're fully migrated)I expect that the Dagster UI is going to service a majority of our existing
argo
CLI use cases, once we migrate. But I want to be sure that I'm accounting for all of our existing processes as we do this migration.What I've found so far
Some
dagster
CLI commands support--grpc-host
and--grpc-port
CLI options. These commands can be run against a port-forward against the Kubernetes user code deployment (not the Dagster instance / webserver). For example, for thedagster job list
command, this works:$ uvx dagster job list --grpc-host localhost --grpc-port 3030 Repository __repository__ ************************* Job: __ASSET_JOB Ops: (Execution Order) hello world
But the CLI docs really don't include much detail on the supported options, or whether they need to be run against a user deployment or the instance/webserver. Since
dagster job list
can be run against a user repository port-forward, I expected thatdagster asset list
could be as well, but it can't -dagster asset list
does not support--grpc-host
/--grpc-port
CLI options. This isn't intuitive to me - I would expect that if the job command can be run this way, that the asset command could be as well.Maybe a better way to do this is via the
dagster-graphql
CLI command? That supports a--remote
CLI option, can I use that with--remote localhost:8080
(if I have the a port-forward open to the dagster-webserver at 8080) ?Beta Was this translation helpful? Give feedback.
All reactions