Add server-backed saved collections for cross-experiment curation
- Dominant language
- Go
- Stars
- 15
- Forks
- 3
- Avg merge
- 21h 53m
- Merged PRs (30d)
- 131
Description
## Problem
Stellar is effective for inspecting and comparing runs within one declared project/experiment/run target, but it does not provide a durable way for a researcher to collect the historical experiments and runs that matter most.
Today, pinned metrics and dashboard layout are target-scoped URL/browser state. Mutable dashboard APIs are intentionally unsupported. A researcher can share a particular view, but cannot maintain a stable collection of baselines, decisive treatments, replications, failed controls, and exact-run evidence across experiments.
This makes Stellar a good viewer but an incomplete research memory.
## Goal
Add a small, generic, server-backed **saved collection** primitive. The UI may call a collection a **Research Board**, but the backend object should remain domain-neutral and store references rather than project-specific scientific interpretation.
A collection should let an authorized user curate targets from multiple experiments, preserve enough view state to render them consistently, and share the result through a stable URL.
## Prior art and product boundary
### Weights & Biases
W&B separates exploratory and durable state:
- Personal workspaces are user-owned sandboxes.
- Saved workspace views are persisted, collaborative snapshots.
- Up to 20 runs can be pinned, including runs from other projects, with one optional baseline.
- Reports provide a durable narrative containing charts and run sets from multiple projects.
- Cross-project comparison remains constrained by compatible metric schemas.
References:
- https://docs.wandb.ai/models/track/workspaces
- https://docs.wandb.ai/models/runs/compare-runs
- https://docs.wandb.ai/models/reports/cross-project-reports
- https://docs.wandb.ai/models/reports/
### Anyscale
Anyscale draws a different boundary. It owns execution observability—cluster health, resource utilization, job state, logs, and Ray workload debugging—but directs users to W&B, TensorBoard, or MLflow for specialized experiment tracking, run comparison, and artifact management.
Reference:
- https://docs.anyscale.com/llm/fine-tuning/observability-and-tracking
The lesson for Stellar is not to recreate W&B Reports. Stellar should natively preserve the exact Tau targets and evidence a researcher considers important, while continuing to support links/integrations to richer external trackers.
## Proposed v1
Introduce a persisted collection with:
- Stable ID, title, optional description, owner/scope, timestamps, and revision metadata.
- Ordered typed references to existing Stellar targets: project, experiment, run group, or run.
- Optional baseline target.
- Selected metrics and section/display configuration needed to reproduce the board view.
- Optional short annotation per referenced target.
- A compatibility/comparison key derived from available experiment contract metadata.
- Stable, shareable URL and authorization inherited from the referenced data/scope.
Add UI actions to:
- Create, rename, and delete a collection.
- Add the current experiment/run target to a collection.
- Remove and reorder entries.
- Mark one entry as the baseline.
- Open a collection and render its entries together.
The collection should store references only. It must not copy run metrics, artifacts, or evidence away from their canonical source.
## Comparison safety
Cross-experiment curation must not imply that every selected target is directly comparable.
- Permit any authorized target to appear as a separate collection entry.
- Overlay or aggregate only targets with compatible metric names, axes, and declared contract/comparison metadata.
- Clearly label incompatible entries and render them side by side rather than silently combining them.
- Preserve links to each target's exact provenance and evidence.
## API shape
Make the server API the source of truth so UI and future automation use the same contract. Avoid a one-off dashboard-only endpoint or a dedicated CLI command that cannot compose with other workflows.
At minimum, expose list/get/create/update/delete operations and add/remove/reorder entry operations. Advertise the capability only when the backing store is writable.
## Acceptance criteria
- A user can create a collection containing targets from at least two experiments.
- Reloading the page or opening its stable URL in another authorized browser reproduces the saved contents and layout.
- A user can add, remove, reorder, annotate, and designate a baseline target.
- References continue to resolve to canonical target data without duplication.
- Unauthorized users cannot infer or access referenced private targets.
- Incompatible targets are retained but cannot be misleadingly overlaid or aggregated.
- API, persistence, capability advertisement, and UI behavior have tests.
- Existing target-scoped URL/localStorage views continue to work as ephemeral exploration state.
## Non-goals
- A collaborative document editor or real-time multiplayer editing.
- W&B-style report publishing, PDF/LaTeX export, or public magic links.
- Reimplementing tracker ingestion, artifact lineage visualization, or arbitrary chart builders.
- Backfilling historical project data as part of this feature.
- Defining project-specific scientific compatibility rules inside Stellar.
Contributor guide
Research direction
The payload names no files, tests, or entry points. Start by tracing the existing target-scoped URL/localStorage state and capability advertisement, then identify the server persistence and API boundaries. Done means authorized users can persist, reload, share, and safely compare cross-experiment collections with API, persistence, capability, and UI tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, database, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100