apache / apache/datafusion-ballista
Store job event logs in an object store so history survives the scheduler
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The event log added in #2264 is hard-wired to the local filesystem. `EventLogWriter` (`ballista/history/src/writer.rs`) opens a `tokio::fs::File` per job and appends JSONL to `/.eventlog`, the reader (`ballista/history/src/reader.rs`) takes a `&Path` and opens it with `std::fs::File::open`, and `HistoryStore::load` (`ballista/scheduler/src/history/mod.rs`) discovers jobs with `std::fs::read_dir`.
That is fine for a single scheduler on a box with a persistent disk, but it is the wrong shape for how Ballista is usually deployed:
- In Kubernetes the scheduler pod's filesystem is ephemeral, so job history dies with the pod unless someone attaches and manages a PVC.
- The history server has to run somewhere that can see the same directory the scheduler wrote to, which in practice means a shared volume rather than a separately deployed read-only service.
- With multiple schedulers there is no single directory holding the whole cluster's history.
Writing logs to an object store (S3, GCS, Azure, or anything else `object_store` supports) removes all three constraints: the scheduler ships logs to a durable remote location, and the history server reads them from anywhere.
**Describe the solution you'd like**
Put the `object_store` crate behind the event log's read and write paths, so `--event-log-dir` accepts a URL (`s3://bucket/prefix`, `file:///var/log/ballista`, plain paths staying local for compatibility):
- `EventLogWriter::new` takes an `Arc` plus a base `Path` instead of a `PathBuf`.
- The reader takes a store and a path rather than `&Path`, and `HistoryStore::load` uses `ObjectStore::list` in place of `read_dir`.
- The history server binary resolves its own store from the same URL, so it can be deployed on its own with no shared volume.
**Describe alternatives you've considered**
- *Keep it local and let operators sync the directory out of band* (sidecar, `aws s3 sync`, PVC + backup). Works, but pushes the durability problem onto every deployment and leaves the history server needing volume access.
- *Trait-abstract the storage layer ourselves* rather than depending on `object_store`. More code for less coverage; `object_store` is already a workspace dependency, already used by `ballista-scheduler`, and already how the rest of the stack talks to remote storage.
**Additional context**
The main design question is append semantics. Object stores have no append: the current writer holds one open handle per job for the life of the process and appends a line per event. Options, roughly in increasing order of effort:
1. **Buffer per job, write once on `JobEnd`.** Simplest, and it fits how the log is consumed today — the reader only looks at the terminal `JobEnd` record and the timeline records are unread. The cost is that a scheduler crash loses the whole job's log, and memory is held for the duration of long-running jobs.
2. **Multipart upload per job**, flushing parts as events accumulate. Preserves incremental durability, but needs careful handling of aborted uploads for jobs whose scheduler dies mid-flight.
3. **Write locally, upload on completion.** Keeps crash-durability on local disk and gets the object into remote storage once the job finishes; needs a cleanup/retry story for uploads that fail.
Worth deciding this before a future UI starts consuming the incremental timeline records, since option 1 rules out live progress from a remote store.
Follow-on from https://github.com/apache/datafusion-ballista/pull/2264#issuecomment-5233620333.
Contributor guide
Research direction
Start with ballista/history/src/writer.rs, ballista/history/src/reader.rs, and ballista/scheduler/src/history/mod.rs, then inspect the existing object_store use in ballista-scheduler. Decide how append semantics should work for the event log before changing the history server's URL resolution. Done means local paths remain compatible while URL-backed stores support writing, reading, and listing history independently of a shared filesystem.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100