apache / apache/datafusion-ballista

Add paging to GET /api/jobs on the scheduler and history server

Open
#2,270 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

`GET /api/jobs` returns every job it knows about in a single response, on both the scheduler (`ballista/scheduler/src/api/handlers.rs`) and the history server added in #2265.

On the scheduler this is bounded in practice, because completed jobs are cleaned up after `finished_job_state_clean_up_interval_seconds` and everything is gone on restart. The history server has no such bound: logs accumulate in the event-log directory until somebody prunes them, so the response grows with however long an operator chooses to retain history. A directory holding tens of thousands of jobs produces a response of a size no client wants, and the TUI refreshes the list periodically.

Raised by @milenkovicm in review on #2265.

**Describe the solution you'd like**

Paging on `/api/jobs`, added to the scheduler and the history server together rather than to one of them.

Doing it on only one side would give the same endpoint two different contracts, and the TUI (`ballista-cli/src/tui/http_client.rs`) points at either one, so it cannot start passing a limit until both understand it.

Sketch, to be settled in the issue rather than assumed:

- Query parameters on `GET /api/jobs`, defaulting to the current behaviour so existing clients keep working.
- A stable order to page over. #2265 makes the history server return jobs newest first by start time, with the job id as a tiebreaker so the order is total. Job ids are random 7-character strings (`TaskManager::generate_job_id`), so they cannot be paged over on their own. The live scheduler currently applies no ordering at all and would need one.
- The TUI asking for a bounded window and requesting more as the user scrolls, instead of re-fetching everything on each refresh.

Worth deciding at the same time whether the response stays a bare JSON array or gains an envelope carrying a total count or a continuation token, since that shape is the part existing clients would notice.

**Describe alternatives you've considered**

- **Leave it unpaged and rely on pruning.** What happens today, and it puts the server's response size at the mercy of an operator's retention policy. It is documented as a limitation on the history server user-guide page, which is worth doing but is not a fix.
- **Cap the response server-side without a paging API.** Silently truncating a list is worse than a large one, since a job that ran would simply not be visible with nothing to say so.
- **Page in the history server only.** Cheaper, but it is the divergence described above.

**Additional context**

The history server side is the more pressing of the two, since it is the one with unbounded retention. See #2265 and the review discussion there.

Contributor guide

Open the contributing guide

Research direction

Start with ballista/scheduler/src/api/handlers.rs, the history server changes from #2265, and ballista-cli/src/tui/http_client.rs. Read the existing GET /api/jobs behavior and TaskManager::generate_job_id, then resolve the ordering, query parameters, response shape, and TUI loading behavior. Done means both servers and the TUI share one paging contract while existing clients retain current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, cli, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.