Custom executor: Run some ops in_process, some ops multiprocess, and group some ops into a shared subprocess

[erenz14]

Question

Does the following seem possible? If so, should I create a custom executor or custom step launcher?

Requirements

• Create a custom executor that can be configured at the op level
• For a single job with multiple ops, you should be able to configure each op differently
  • in_process: any op can run in the parent process
  • multiprocess: any op can run in its own newly created subprocess
  • grouped_multiprocess: groups of ops can run in a shared subprocess (diagram attached)

Use case

• If an op is not running in parallel then we would prefer to run it in the parent process, to save the overhead of creating a new subprocess. For this op, we'd use in_process.
• If an op is running in parallel for multiple DynamicOutput, then we'd want to create a single subprocess for each branch, but allow the subprocess to be shared for downstream ops within that same branch (diagram attached). As shown in the attached diagram, for the 3 branches this would allow us to only create 3 subprocesses (1 for each branch) as opposed to 6 (1 for each op)

[dmsfabiano]

Also interested in this functionality!

[david-takacs]

hi @erenz14 did you end up with a solution for achieving this kind of behavior?

[erenz14]

No we did not. We gave up on the ideal approach, and just ended up hosting our ML models as a separate service API and then within a single op we call multiple APIs in parallel using asyncio (could also using multithreading).