[Feature] datasets fs: inspect and download files under a task (ls/get/download)
- Dominant language
- Python
- Stars
- 485
- Forks
- 81
- Avg merge
- 16h 12m
- Merged PRs (30d)
- 8
Description
**Feature Category**
- [ ] Sandbox
- [ ] Actions
- [ ] Deployments
- [x] SDK & API
- [ ] Envhub
- [x] CLI
- [ ] Performance & Optimization
- [ ] Documentation & Examples
**Problem Statement**
There is currently no way to inspect or fetch individual files inside a single
task from the OSS dataset registry. Users can list task IDs (`datasets tasks`)
but to look at or download a task's contents they must reach for raw `ossutil`
and reconstruct the `datasets/{org}/{dataset}/{split}/{task}/` key layout by
hand. Two concrete gaps:
1. **No file listing per task**: no command answers "what files does task X
contain?"
2. **No file read / download**: no command prints a single task file to stdout
or downloads a task file/directory to a local path.
**Proposed Solution**
Add a `datasets fs` subcommand group (alias `files`) with three operations,
backed by new registry/client methods.
1. **Registry layer** (`OssDatasetRegistry` / `BaseDatasetRegistry` /
`DatasetClient`):
- `list_task_files(org, dataset, split, task_id, path="") -> list[TaskFile]`
— list files under a task, paths relative to the task root.
- `get_task_file(org, dataset, split, task_id, path) -> bytes | None`
— read one task file by relative path; `None` when the object is absent.
- New `TaskFile` dataclass (`path: str`, `size: int | None`).
- Support both layouts: directory-style tasks (`{task}/...`) and
single-file tasks (`{task}.json` directly under the split).
2. **CLI layer** (`datasets fs ...`):
- `datasets fs ls --org --dataset --split --task [--path]` — list files.
- `datasets fs get --org --dataset --split --task [--path]` — print one
file to stdout (resolves the only file automatically when the task has
exactly one).
- `datasets fs download --org --dataset --split --task --path --dest` —
download a single file or a whole directory subtree to a local path.
- JSON output (`-o json`) for `ls`/`get`/`download`.
3. **Path safety**: relative task paths are normalized and validated — reject
absolute paths and `..` traversal, drop `.`/empty segments — so a crafted
`--path` cannot escape the task root.
**Notes**
- Split out from #1063 / PR #1064: #1063 covers the `datasets tasks`
pagination/cache/`--filter` work; this issue tracks the independent
`datasets fs` file-access feature.
- The CLI command module (`rock/cli/command/datasets.py`) will be refactored
toward a clearer object-oriented structure as part of this change (per review
feedback on PR #1064): path normalization and output handling extracted into
dedicated `TaskPath` / `OutputWriter` helpers, with the module-level helper
functions folded into the command class.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with rock/cli/command/datasets.py and trace the OssDatasetRegistry, BaseDatasetRegistry, and DatasetClient entry points mentioned in the issue. Review the existing datasets tasks command and PR #1064 context before defining the fs command structure. Done means ls, get, and download support both task layouts, safe relative paths, and JSON output as specified, with registry methods and TaskFile added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100