[dagster-sigma] Add dagster-sigma snapshot CLI, which allows persisting the contents of a Sigma workspace to a file

[benpankow]

Summary

Introduces a utility CLI for dagster-sigma which loads a code location containing one or more SigmaOrganizationDefsLoaders and serializes the repository load data to a file. Adds the capability for the Sling defs loader to ingest this snapshot from a file, allowing orgs with large amounts of Sigma data to ingest at CI time or manually vs each code location load.

In the future, we expect to subsume this behavior into a framework level solution which doesn't tie loading external state into code location load. For now, this is a bit of a workaround.

dagster-sigma snapshot --python-module my_dagster_package --save-to snapshot.snap

...
specs = load_sigma_asset_specs(sigma_workspace, snapshot_path=Path(__file__).parent / "snapshot.snap")
...

How I Tested These Changes

Unit test, more forthcoming.

Changelog

[dagster-sigma] Introduced dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

[benpankow]

• [dagster-sigma] Add dagster-sigma snapshot CLI, which allows persisting the contents of a Sigma workspace to a file #25913 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @benpankow and the rest of your teammates on Graphite