apache / apache/datafusion-ballista

Cancel tasks in query stage early if limit has been reached

Open
#628 0 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 11h
Merged PRs (30d)
86

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In presence of a limit, Ballista could cancel / remove the remaining (both active/inactive) tasks once the total number of rows exceeds the limit.

**Describe the solution you'd like**
For each stage, add an optional `limit`. The query stage scheduler could track the total number of rows, based on the task metrics
and update the number of rows accordingly. It could cancel the remaining tasks and continue with the next stage.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the query-stage scheduler and the task metrics used to count rows. The work is complete when stages can accept an optional limit, row totals are tracked, remaining active and inactive tasks are canceled after the limit is reached, and execution continues with the next stage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.