apache / apache/datafusion-ballista
Remove sleep from pull-based event loop
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The current implementation sleeps (currently ~100ms) whenever there is no job active. This adds some extra latency to the jobs.
We could instead have the scheduler send a message to the executor that a new tasks are available.
**Describe the solution you'd like**
have the scheduler send a message to the executor that tasks are available.
remove sleeping logic from `poll_loop` (e.g. implement with mutex)
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by locating the executor's `poll_loop` and the scheduler-to-executor task-availability path. Determine how that communication can replace the current sleep; done means the executor no longer sleeps when idle and newly available tasks are handled without the added latency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100