apache / apache/gravitino

[EPIC] Improve the job system: output logs, runtime job template, and DB-layer optimizations

Open
#12,667 0 comments 0 reactions 1 assignee Claimed by @jerryshao View on GitHub
epic
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the proposal

Follow-up hardening for the job system (`api/.../job`, `core/.../job`, `core/.../storage/relational/service/JobMetaService.java`) covering three independent tracks:

1. **Job output log support** — `JobExecutor` (`core/src/main/java/org/apache/gravitino/connector/job/JobExecutor.java`) currently only exposes `submitJob`/`getJobStatus`/`cancelJob`; there is no way to retrieve a job's stdout/stderr. Add a way for `JobExecutor` implementations (local runner, Airflow, etc.) to surface job output, and expose it through `JobHandle` and the REST/client APIs, following the same pattern used for `queuedAt`/`startedAt`/`finishedAt` (default method throwing `UnsupportedOperationException` for backward compatibility).
2. **Runtime job template on `JobHandle`** — today `JobTemplate` placeholders are resolved before being handed to `JobExecutor#submitJob`, but the resolved (runtime) template isn't returned anywhere; callers only ever see the original template. Expose the actual runtime template that was submitted via `JobHandle`.
3. **DB logic optimizations** — review `JobMetaService`/`JobTemplateMetaService` and their SQL providers for the same class of issues recently fixed in the fileset OCC work (e.g. #12656): unnecessary round trips, soft-delete/cleanup logic (`JobMetaBaseSQLProvider.softDeleteJobMetasByLegacyTimeline`), and any missing optimistic-concurrency protection on job/job-template updates.

### Task list

- [ ] Support output log retrieval for job handler (`JobExecutor`, `JobHandle`, REST + Java/Python clients)
- [ ] #12716
- [ ] Expose the runtime (placeholder-resolved) job template via `JobHandle`
- [x] #12676
- [ ] Audit and optimize `JobMetaService` / `JobTemplateMetaService` DB access patterns
- [x] #12669

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.