Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DISCUSSION] Add DedicatedExecutor into the DataFusion crates to make using multiple threadpools easier #14285

Open
alamb opened this issue Jan 24, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Jan 24, 2025

Is your feature request related to a problem or challenge?

Describe the solution you'd like

As explained on the following PR:

I implemented a version of DedicatedExecutor and an example showing how to use multiple thread pools with DataFusion. However, the example is currently very large as it uses an abstraction called DedicatedExecutor to manage sending work back and forth between the runtimes

If users want to do the same thing, they will need to copy DedicatedExecutor into their code (or make something similar).

Describe alternatives you've considered

If DedicatedExecutor is a useful API for multiple people I think we should consider moving it into one of the core crates

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant