[DISCUSSION] Add `DedicatedExecutor` into the DataFusion crates to make using multiple threadpools easier
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
- Related to https://github.com/apache/datafusion/issues/12393
### Describe the solution you'd like
As explained on the following PR:
- https://github.com/apache/datafusion/pull/14286
I implemented a version of `DedicatedExecutor` and an example showing how to use multiple thread pools with DataFusion. However, the example is currently very large as it uses an abstraction called `DedicatedExecutor` to manage sending work back and forth between the runtimes
If users want to do the same thing, they will need to copy `DedicatedExecutor` into their code (or make something similar).
### Describe alternatives you've considered
If `DedicatedExecutor` is a useful API for multiple people I think we should consider moving it into one of the core crates
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the related issue 12393 and pull request 14286, which contain the current DedicatedExecutor implementation and example. Determine whether the abstraction belongs in a DataFusion core crate and what API maintainers want; the work is done when the crate placement and public API are agreed and implemented with appropriate coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100