how do I get graph outs to pick up the data types of the outs returned by the inner ops? #18009
Unanswered
Replies: 1 comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
how do I get graph outs to pick up the data types of the outs returned by the inner ops? 🧵
seems like the graph is unable to infer types from the ops contained within it, and I don't see any configuration available on GraphOut to explicitly define the types:
@graph(
out={"return_code": GraphOut(), "target_path": GraphOut(), "artifacts": GraphOut(), "end": GraphOut()},
tags={"kind": "dbt"},
name=name,
)
def _graph(start: Nothing):
return_code, target_path = op(start)
artifacts = configured_unload_artifacts_v2(target_path)
end = handle_dbt_return_code(return_code)
return {"return_code": return_code, "target_path": target_path, "artifacts": artifacts, "end": end}
Out of curiosity I did some digging into the graph implementation and found this (graph_decorator.py#61)
if self.out is None:
output_defs = self.output_defs
elif isinstance(self.out, GraphOut):
output_defs = [self.out.to_definition(name=None)]
else:
check.dict_param(self.out, "out", key_type=str, value_type=GraphOut)
output_defs = [out.to_definition(name=name) for name, out in self.out.items()]
and then also checked the implementation of GraphOut and saw that it’s VERY brief (output.py#499)
class GraphOut(NamedTuple("_GraphOut", [("description", PublicAttr[Optional[str]])])):
"""Represents information about the outputs that a graph maps.
and it doesn’t look like it does any annotation passing or even let you specify it manually. For comparison, the Out class has the following in its to_definition implementation to use an explicitly specified type if available, and try to resolve from annotations if not
dagster_type = (
self.dagster_type
if self.dagster_type is not NoValueSentinel
else _checked_inferred_type(annotation_type)
)
because this is missing from GraphOut I would wager it’s not possible, but I’m also interested in seeing what a Dagster employee has to say
The question was originally asked in Dagster Slack.
Beta Was this translation helpful? Give feedback.
All reactions