[dagster-airlift][federation-tutorial] Observe section federation tutorial #25892
+411
−7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 13, 2024
Merged
This was referenced Nov 13, 2024
Graphite Automations"Add a 'docs-to-migrate' label to PRs with docs" took an action on this PR • (11/13/24)1 label was added to this PR based on Christopher DeCarolis's automation. |
cmpadden
reviewed
Nov 13, 2024
docs/content/_navigation.json
Outdated
| @@ -949,6 +949,10 @@ | |||
| { | |||
| "title": "Part 1: Setup upstream and downstream Airflow instances", | |||
| "path": "/integrations/airlift/federation-tutorial/setup" | |||
| }, | |||
| { | |||
| "title": "Part 2: Observe dag lineage in Dagster", | |||
There was a problem hiding this comment.
Suggested change
| "title": "Part 2: Observe dag lineage in Dagster", | |
| "title": "Part 2: Observe DAG lineage in Dagster", |
cmpadden
reviewed
Nov 13, 2024
| @@ -0,0 +1,208 @@ | |||
| # Airflow Federation Tutorial: Observing multiple Airflow instances | |||
|
|
|||
| At this point, we should have finished the [setup](/integrations/airlift/federation-tutorial/setup) step, and now we have the example code setup with a fresh virtual environment, and our two Airflow instances running locally. Now, we can start writing Dagster code. | |||
There was a problem hiding this comment.
Suggested change
| At this point, we should have finished the [setup](/integrations/airlift/federation-tutorial/setup) step, and now we have the example code setup with a fresh virtual environment, and our two Airflow instances running locally. Now, we can start writing Dagster code. | |
| At this point, you should have finished the [setup](/integrations/airlift/federation-tutorial/setup) step, and now you have the example code in a fresh virtual environment, and two Airflow instances running locally. Now, let's start writing Dagster code. |
cmpadden
reviewed
Nov 13, 2024
|
|
||
| ## Observing the Airflow instances | ||
|
|
||
| We'll start by creating asset representations of our DAGs in Dagster. |
There was a problem hiding this comment.
Suggested change
| We'll start by creating asset representations of our DAGs in Dagster. | |
| We'll start by creating asset representations of our Airflow DAGs in Dagster. |
cmpadden
reviewed
Nov 13, 2024
|
|
||
| ### Observing the `warehouse` Airflow instance | ||
|
|
||
| Next, we'll declare a reference to our `warehouse` Airflow instance, which is running at `http://localhost:8081`. |
There was a problem hiding this comment.
I requested changes above of we -> you, but let's just make sure there's consistency across guides.
cmpadden
approved these changes
Nov 13, 2024
dpeng817
changed the title
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
fc3bfd8
to
56dc056
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
ea3067a
to
2390f28
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
56dc056
to
e4270a2
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
2390f28
to
d038556
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
e4270a2
to
5e67354
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
d038556
to
3eb8689
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
2 times, most recently
from
2ac9632
to
13ff868
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
3eb8689
to
9b8354a
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
13ff868
to
09b59d8
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
9b8354a
to
2adfd07
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| ) | ||
| # end_lineage | ||
|
|
||
| defs = Definitions(assets=[load_customers, customer_metrics_dag_asset]) |
There was a problem hiding this comment.
The final Definitions object is missing the sensor registrations that were defined earlier. To ensure both DAGs are properly monitored, the sensors should be included:
defs = Definitions(
assets=[load_customers, customer_metrics_dag_asset],
sensors=[warehouse_sensor, metrics_sensor]
)This ensures that both the warehouse and metrics DAGs will be monitored for new runs.
Spotted by Graphite Reviewer
Is this helpful? React 👍 or 👎 to let us know.
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
09b59d8
to
ee91104
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
2adfd07
to
71aa819
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
ee91104
to
64722c3
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
71aa819
to
72e4fe5
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
64722c3
to
b896f9c
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
72e4fe5
to
48e9247
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
b896f9c
to
1821ae4
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
48e9247
to
86ce701
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
1821ae4
to
cb5258e
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
86ce701
to
3940172
Compare
dpeng817
force-pushed
the
dpeng817/implement_dags
branch
from
cb5258e
to
31e1b1e
Compare
dpeng817
force-pushed
the
dpeng817/fed-tutorial-observe-section
branch
from
3940172
to
7da4baa
Compare
Base automatically changed from
dpeng817/implement_dags
to
dpeng817/federation-tutorial
November 14, 2024 18:10
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
How I Tested These Changes
Changelog