Azure / Azure/taugrid

Add server-backed saved collections for cross-experiment curation

Open
#83 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
15
Forks
3
Avg merge
21h 53m
Merged PRs (30d)
131

Description

## Problem

Stellar is effective for inspecting and comparing runs within one declared project/experiment/run target, but it does not provide a durable way for a researcher to collect the historical experiments and runs that matter most.

Today, pinned metrics and dashboard layout are target-scoped URL/browser state. Mutable dashboard APIs are intentionally unsupported. A researcher can share a particular view, but cannot maintain a stable collection of baselines, decisive treatments, replications, failed controls, and exact-run evidence across experiments.

This makes Stellar a good viewer but an incomplete research memory.

## Goal

Add a small, generic, server-backed **saved collection** primitive. The UI may call a collection a **Research Board**, but the backend object should remain domain-neutral and store references rather than project-specific scientific interpretation.

A collection should let an authorized user curate targets from multiple experiments, preserve enough view state to render them consistently, and share the result through a stable URL.

## Prior art and product boundary

### Weights & Biases

W&B separates exploratory and durable state:

- Personal workspaces are user-owned sandboxes.
- Saved workspace views are persisted, collaborative snapshots.
- Up to 20 runs can be pinned, including runs from other projects, with one optional baseline.
- Reports provide a durable narrative containing charts and run sets from multiple projects.
- Cross-project comparison remains constrained by compatible metric schemas.

References:

- https://docs.wandb.ai/models/track/workspaces
- https://docs.wandb.ai/models/runs/compare-runs
- https://docs.wandb.ai/models/reports/cross-project-reports
- https://docs.wandb.ai/models/reports/

### Anyscale

Anyscale draws a different boundary. It owns execution observability—cluster health, resource utilization, job state, logs, and Ray workload debugging—but directs users to W&B, TensorBoard, or MLflow for specialized experiment tracking, run comparison, and artifact management.

Reference:

- https://docs.anyscale.com/llm/fine-tuning/observability-and-tracking

The lesson for Stellar is not to recreate W&B Reports. Stellar should natively preserve the exact Tau targets and evidence a researcher considers important, while continuing to support links/integrations to richer external trackers.

## Proposed v1

Introduce a persisted collection with:

- Stable ID, title, optional description, owner/scope, timestamps, and revision metadata.
- Ordered typed references to existing Stellar targets: project, experiment, run group, or run.
- Optional baseline target.
- Selected metrics and section/display configuration needed to reproduce the board view.
- Optional short annotation per referenced target.
- A compatibility/comparison key derived from available experiment contract metadata.
- Stable, shareable URL and authorization inherited from the referenced data/scope.

Add UI actions to:

- Create, rename, and delete a collection.
- Add the current experiment/run target to a collection.
- Remove and reorder entries.
- Mark one entry as the baseline.
- Open a collection and render its entries together.

The collection should store references only. It must not copy run metrics, artifacts, or evidence away from their canonical source.

## Comparison safety

Cross-experiment curation must not imply that every selected target is directly comparable.

- Permit any authorized target to appear as a separate collection entry.
- Overlay or aggregate only targets with compatible metric names, axes, and declared contract/comparison metadata.
- Clearly label incompatible entries and render them side by side rather than silently combining them.
- Preserve links to each target's exact provenance and evidence.

## API shape

Make the server API the source of truth so UI and future automation use the same contract. Avoid a one-off dashboard-only endpoint or a dedicated CLI command that cannot compose with other workflows.

At minimum, expose list/get/create/update/delete operations and add/remove/reorder entry operations. Advertise the capability only when the backing store is writable.

## Acceptance criteria

- A user can create a collection containing targets from at least two experiments.
- Reloading the page or opening its stable URL in another authorized browser reproduces the saved contents and layout.
- A user can add, remove, reorder, annotate, and designate a baseline target.
- References continue to resolve to canonical target data without duplication.
- Unauthorized users cannot infer or access referenced private targets.
- Incompatible targets are retained but cannot be misleadingly overlaid or aggregated.
- API, persistence, capability advertisement, and UI behavior have tests.
- Existing target-scoped URL/localStorage views continue to work as ephemeral exploration state.

## Non-goals

- A collaborative document editor or real-time multiplayer editing.
- W&B-style report publishing, PDF/LaTeX export, or public magic links.
- Reimplementing tracker ingestion, artifact lineage visualization, or arbitrary chart builders.
- Backfilling historical project data as part of this feature.
- Defining project-specific scientific compatibility rules inside Stellar.

Contributor guide

Open the contributing guide

Research direction

The payload names no files, tests, or entry points. Start by tracing the existing target-scoped URL/localStorage state and capability advertisement, then identify the server persistence and API boundaries. Done means authorized users can persist, reload, share, and safely compare cross-experiment collections with API, persistence, capability, and UI tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, database, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.