[Feature] Unify job run orchestration for single, multi-task, and full dataset runs
- Dominant language
- Python
- Stars
- 485
- Forks
- 81
- Avg merge
- 16h 12m
- Merged PRs (30d)
- 8
Description
## Feature Category
- SDK & API
- CLI
- Envhub
- Documentation & Examples
## Problem Statement
`rock job run` currently focuses on running a single sandbox job. Dataset-scale workflows need users or upper-level wrappers to manually enumerate tasks, submit jobs one by one, track local progress, and recover interrupted runs.
Key gaps:
1. No unified CLI entrypoint for single-task, multi-task, and full dataset execution.
2. Full dataset execution should remain a CLI-level orchestration capability, while the SDK should only expose reusable atomic primitives.
3. Interrupted local processes cannot reliably resume a run and reconnect to still-running sandbox jobs.
4. There is no run-level metadata that maps `run_id -> task_id -> job_name`, making status/show queries difficult.
5. Upper-level wrappers need JSONL progress events for automation.
## Proposed Solution
1. Extend `rock job run` as the unified entrypoint:
- `--task ` for one explicit task
- `--tasks ` for selected tasks
- `--all` for full dataset split execution
- `--resume ` for interrupted runs
- `--concurrency`, `--limit`, and `--jsonl` for execution control and automation
2. Keep SDK/CLI responsibilities separated:
- SDK provides atomic job planning, job execution, metadata repository, and reconnect/wait primitives.
- CLI composes these primitives into single, multi-task, and full dataset orchestration.
3. Add two-level metadata:
- Run metadata under the experiment directory, recording `run_id`, mode, task list, task/job mapping, status, and score summary.
- Job metadata under each job directory, recording sandbox id, session, pid, task id, status, score, error, and timestamps.
4. Support resume semantics:
- Skip completed tasks.
- Reconnect to still-running sandbox jobs when `sandbox_id/session/pid` are available.
- Mark unrecoverable previous attempts and rerun when sandbox/process metadata is missing or unusable.
5. Add run-level query commands:
- `rock job runs`
- `rock job status --run-id `
- `rock job show --run-id --task-id `
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing `rock job run` CLI entrypoint and identify the SDK primitives currently available for planning, execution, metadata, and reconnect/wait behavior. Review how experiment and job directories store metadata, then define the run, status, resume, and JSONL behavior described here; done means single-task, selected-task, full-dataset, and interrupted-run workflows are covered by the relevant CLI and SDK tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100