apache / apache/datafusion-ballista

history server event log should capture scheduler id

Open
#2,413 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

at the moment event logs are written in local directory, so associating logs with scheduler is not hard task, if we ship logs to a centralised location (s3 bucket) being able to associate logs with schedulers would help with diagnostics. current assumption is that shipping event logs to s3 would be done by external process

**Describe the solution you'd like**

add additional field to `JobStart` event, like `scheduler_id`

change from

```json
{
"ev": "JobStart",
"version": 1,
"data": {
"job_id": "0382W17Y0G000",
"job_name": "Query derived from TPC-H q1",
"queued_at": 1788250553353,
"submitted_at": 1788250553364,
```
to

```json
{
"ev": "JobStart",
"version": 1,
"data": {
"job_id": "0382W17Y0G000",
"job_name": "Query derived from TPC-H q1",
"queued_at": 1788250553353,
"submitted_at": 1788250553364,
"scheduler_id" : "< random scheduler id >"
```

**Describe alternatives you've considered**

alternative would be to have a path prefix per scheduler id, which at the moment we cant retrieve from available apis

```bash
curl http://localhost:50050/api/state
{"started":1788384057783,"version":"54.0.0","datafusion_version":"55.0.0","substrait_support":false,"keda_support":false,"prometheus_support":false,"graphviz_support":false,"spark_support":false,"scheduling_policy":"push-staged","enable_embedded_flight_proxy":false}%
```

**Additional context**

we do not have scheduler id associated with the scheduler, so we might need to add it

Contributor guide

Open the contributing guide

Research direction

Start by tracing the JobStart event and the scheduler state exposed by the /api/state endpoint. Determine where a scheduler identity can be created and how the event log is serialized; done means JobStart records that identity consistently and the resulting event can associate centralized logs with a scheduler.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.