apache / apache/datafusion-ballista
history server event log should capture scheduler id
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
at the moment event logs are written in local directory, so associating logs with scheduler is not hard task, if we ship logs to a centralised location (s3 bucket) being able to associate logs with schedulers would help with diagnostics. current assumption is that shipping event logs to s3 would be done by external process
**Describe the solution you'd like**
add additional field to `JobStart` event, like `scheduler_id`
change from
```json
{
"ev": "JobStart",
"version": 1,
"data": {
"job_id": "0382W17Y0G000",
"job_name": "Query derived from TPC-H q1",
"queued_at": 1788250553353,
"submitted_at": 1788250553364,
```
to
```json
{
"ev": "JobStart",
"version": 1,
"data": {
"job_id": "0382W17Y0G000",
"job_name": "Query derived from TPC-H q1",
"queued_at": 1788250553353,
"submitted_at": 1788250553364,
"scheduler_id" : "< random scheduler id >"
```
**Describe alternatives you've considered**
alternative would be to have a path prefix per scheduler id, which at the moment we cant retrieve from available apis
```bash
curl http://localhost:50050/api/state
{"started":1788384057783,"version":"54.0.0","datafusion_version":"55.0.0","substrait_support":false,"keda_support":false,"prometheus_support":false,"graphviz_support":false,"spark_support":false,"scheduling_policy":"push-staged","enable_embedded_flight_proxy":false}%
```
**Additional context**
we do not have scheduler id associated with the scheduler, so we might need to add it
Contributor guide
Research direction
Start by tracing the JobStart event and the scheduler state exposed by the /api/state endpoint. Determine where a scheduler identity can be created and how the event log is serialized; done means JobStart records that identity consistently and the resulting event can associate centralized logs with a scheduler.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100