You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Asset A and Asset B are both partitioned by month.
I materialize Asset A every minute to fetch data, and whenever there is a change in this data, I need to update the downstream asset, Asset B.
Additionally, I occasionally update Asset A's backfill, and similarly, I need to instruct the downstream asset to perform a backfill for the same partition key.
My current solution
When I materialize Asset A, I store a hash of the DataVersion as a filename in an empty file (using AssetKey and partition key as the directory path).
I created a sensor that continuously checks for new files in the AssetKey directory of Asset A. If the number of files increases, I use a RunRequest to update the corresponding partition of Asset B.
My question
I would like to refactor this process to utilize more of Dagster's built-in features. I learned about AutomationCondition.data_version_changed(), but there doesn't seem to be detailed documentation on it.
I also tried using observable_source_asset, but when defining the asset, I encountered an error in the UI when creating an observation, as it did not include the partition key. Additionally, I noticed that SourceAsset is set to be deprecated, so I'm unsure whether I should use it.
Are there any other solutions that can help me accomplish my task?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Task I want to accomplish
Asset A and Asset B are both partitioned by month.
I materialize Asset A every minute to fetch data, and whenever there is a change in this data, I need to update the downstream asset, Asset B.
Additionally, I occasionally update Asset A's backfill, and similarly, I need to instruct the downstream asset to perform a backfill for the same partition key.
My current solution
When I materialize Asset A, I store a hash of the DataVersion as a filename in an empty file (using AssetKey and partition key as the directory path).
I created a sensor that continuously checks for new files in the AssetKey directory of Asset A. If the number of files increases, I use a RunRequest to update the corresponding partition of Asset B.
My question
I would like to refactor this process to utilize more of Dagster's built-in features. I learned about
AutomationCondition.data_version_changed()
, but there doesn't seem to be detailed documentation on it.I also tried using
observable_source_asset
, but when defining the asset, I encountered an error in the UI when creating an observation, as it did not include the partition key. Additionally, I noticed thatSourceAsset
is set to be deprecated, so I'm unsure whether I should use it.Are there any other solutions that can help me accomplish my task?
Beta Was this translation helpful? Give feedback.
All reactions