alibaba / alibaba/ROCK

[Feature] Unify job run orchestration for single, multi-task, and full dataset runs

Open
#1,234 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
485
Forks
81
Avg merge
16h 12m
Merged PRs (30d)
8

Description

## Feature Category

- SDK & API
- CLI
- Envhub
- Documentation & Examples

## Problem Statement

`rock job run` currently focuses on running a single sandbox job. Dataset-scale workflows need users or upper-level wrappers to manually enumerate tasks, submit jobs one by one, track local progress, and recover interrupted runs.

Key gaps:

1. No unified CLI entrypoint for single-task, multi-task, and full dataset execution.
2. Full dataset execution should remain a CLI-level orchestration capability, while the SDK should only expose reusable atomic primitives.
3. Interrupted local processes cannot reliably resume a run and reconnect to still-running sandbox jobs.
4. There is no run-level metadata that maps `run_id -> task_id -> job_name`, making status/show queries difficult.
5. Upper-level wrappers need JSONL progress events for automation.

## Proposed Solution

1. Extend `rock job run` as the unified entrypoint:
- `--task ` for one explicit task
- `--tasks ` for selected tasks
- `--all` for full dataset split execution
- `--resume ` for interrupted runs
- `--concurrency`, `--limit`, and `--jsonl` for execution control and automation

2. Keep SDK/CLI responsibilities separated:
- SDK provides atomic job planning, job execution, metadata repository, and reconnect/wait primitives.
- CLI composes these primitives into single, multi-task, and full dataset orchestration.

3. Add two-level metadata:
- Run metadata under the experiment directory, recording `run_id`, mode, task list, task/job mapping, status, and score summary.
- Job metadata under each job directory, recording sandbox id, session, pid, task id, status, score, error, and timestamps.

4. Support resume semantics:
- Skip completed tasks.
- Reconnect to still-running sandbox jobs when `sandbox_id/session/pid` are available.
- Mark unrecoverable previous attempts and rerun when sandbox/process metadata is missing or unusable.

5. Add run-level query commands:
- `rock job runs`
- `rock job status --run-id `
- `rock job show --run-id --task-id `

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing `rock job run` CLI entrypoint and identify the SDK primitives currently available for planning, execution, metadata, and reconnect/wait behavior. Review how experiment and job directories store metadata, then define the run, status, resume, and JSONL behavior described here; done means single-task, selected-task, full-dataset, and interrupted-run workflows are covered by the relevant CLI and SDK tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.