apache / apache/datafusion-ballista

Store job event logs in an object store so history survives the scheduler

Open
#2,267 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The event log added in #2264 is hard-wired to the local filesystem. `EventLogWriter` (`ballista/history/src/writer.rs`) opens a `tokio::fs::File` per job and appends JSONL to `/.eventlog`, the reader (`ballista/history/src/reader.rs`) takes a `&Path` and opens it with `std::fs::File::open`, and `HistoryStore::load` (`ballista/scheduler/src/history/mod.rs`) discovers jobs with `std::fs::read_dir`.

That is fine for a single scheduler on a box with a persistent disk, but it is the wrong shape for how Ballista is usually deployed:

- In Kubernetes the scheduler pod's filesystem is ephemeral, so job history dies with the pod unless someone attaches and manages a PVC.
- The history server has to run somewhere that can see the same directory the scheduler wrote to, which in practice means a shared volume rather than a separately deployed read-only service.
- With multiple schedulers there is no single directory holding the whole cluster's history.

Writing logs to an object store (S3, GCS, Azure, or anything else `object_store` supports) removes all three constraints: the scheduler ships logs to a durable remote location, and the history server reads them from anywhere.

**Describe the solution you'd like**

Put the `object_store` crate behind the event log's read and write paths, so `--event-log-dir` accepts a URL (`s3://bucket/prefix`, `file:///var/log/ballista`, plain paths staying local for compatibility):

- `EventLogWriter::new` takes an `Arc` plus a base `Path` instead of a `PathBuf`.
- The reader takes a store and a path rather than `&Path`, and `HistoryStore::load` uses `ObjectStore::list` in place of `read_dir`.
- The history server binary resolves its own store from the same URL, so it can be deployed on its own with no shared volume.

**Describe alternatives you've considered**

- *Keep it local and let operators sync the directory out of band* (sidecar, `aws s3 sync`, PVC + backup). Works, but pushes the durability problem onto every deployment and leaves the history server needing volume access.
- *Trait-abstract the storage layer ourselves* rather than depending on `object_store`. More code for less coverage; `object_store` is already a workspace dependency, already used by `ballista-scheduler`, and already how the rest of the stack talks to remote storage.

**Additional context**

The main design question is append semantics. Object stores have no append: the current writer holds one open handle per job for the life of the process and appends a line per event. Options, roughly in increasing order of effort:

1. **Buffer per job, write once on `JobEnd`.** Simplest, and it fits how the log is consumed today — the reader only looks at the terminal `JobEnd` record and the timeline records are unread. The cost is that a scheduler crash loses the whole job's log, and memory is held for the duration of long-running jobs.
2. **Multipart upload per job**, flushing parts as events accumulate. Preserves incremental durability, but needs careful handling of aborted uploads for jobs whose scheduler dies mid-flight.
3. **Write locally, upload on completion.** Keeps crash-durability on local disk and gets the object into remote storage once the job finishes; needs a cleanup/retry story for uploads that fail.

Worth deciding this before a future UI starts consuming the incremental timeline records, since option 1 rules out live progress from a remote store.

Follow-on from https://github.com/apache/datafusion-ballista/pull/2264#issuecomment-5233620333.

Contributor guide

Open the contributing guide

Research direction

Start with ballista/history/src/writer.rs, ballista/history/src/reader.rs, and ballista/scheduler/src/history/mod.rs, then inspect the existing object_store use in ballista-scheduler. Decide how append semantics should work for the event log before changing the history server's URL resolution. Done means local paths remain compatible while URL-backed stores support writing, reading, and listing history independently of a shared filesystem.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.