apache / apache/datafusion-ballista
Cancel tasks in query stage early if limit has been reached
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 86
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In presence of a limit, Ballista could cancel / remove the remaining (both active/inactive) tasks once the total number of rows exceeds the limit.
**Describe the solution you'd like**
For each stage, add an optional `limit`. The query stage scheduler could track the total number of rows, based on the task metrics
and update the number of rows accordingly. It could cancel the remaining tasks and continue with the next stage.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by locating the query-stage scheduler and the task metrics used to count rows. The work is complete when stages can accept an optional limit, row totals are tracked, remaining active and inactive tasks are canceled after the limit is reached, and execution continues with the next stage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100