Unify entity-walking code paths; make entity collation version-aware

Follow-up to PR #283 / #284 / #285 capturing the design discussion at https://github.com/Medical-Event-Data-Standard/MEDS-DEV/pull/283#discussion_r3219530904.

Two related design changes deferred so the current PRs could land their building-block scope:

(a) Unify the entity-walking code paths

Today two places walk the same directory tree:

1. The registries — MEDS_DEV.datasets.__init__, MEDS_DEV.tasks.__init__, MEDS_DEV.models.__init__. Each ~5 lines of rglob + relative-path keying, producing typed Python objects (DatasetMetadata/TaskMetadata/Metadata dataclasses plus Path references). Used at runtime.
2. MEDS_DEV.web.collate_entities — walks the same paths and produces JSON-serializable tree dicts that include README content, refs.bib content, and category-parent structure. Used by the regen workflow to build _web/entities/*.json.

Real overlap: both glob the same patterns (datasets/**/dataset.yaml, tasks/**/*.yaml, models/**/model.yaml), both key entries by relative_to(...).with_suffix("").as_posix().

Distinct: the registries produce typed objects for runtime use; collate_entities produces serialization-ready dicts and walks additional auxiliary files (READMEs, refs.bib, category-parent READMEs). The registries also use importlib.resources.files("MEDS_DEV.<X>") (installed-package targeting); collate_entities takes a --repo_dir and can run against any source tree.

Options discussed in #283:

• Currently shipping: keep them separate + add a coherence test that asserts set(collate_entities(repo)) ⊇ DATASETS.keys() ∪ TASKS.keys() ∪ MODELS.keys().
• Option 2: have collate_entities iterate the registries directly. Cleanest source-of-truth story; cost is tying serialization to runtime imports and the installed-package model.
• Option 3: extract a shared walker utility. Marginal — each call site is small.
• Option 4: push the auxiliary content (readme/refs/category-tree) into the registries themselves. Big blast radius across every consumer.

(b) Make entity collation version-aware

Today the regen workflow (#285) is triggered on every push to main and writes a single set of entities/{datasets,tasks,models}.json to the _web branch. That's the "live" model — the website always reflects current main.

Goal: transition to a "release-snapshot" model where entity manifests are captured at release tags and stored in a versioned tree, so the website can navigate historical states (matching how result blobs already carry a version field).

Proposed shape:

• The CLI stays as a pure tree → JSON function.
• The regen workflow's trigger pivots from push: branches: [main] → push: tags: ['v*'].
• _web/entities/ layout becomes versioned:

  _web/entities/
  ├── index.json          # {"versions": [...], "latest": "..."}
  ├── 0.0.15/{datasets,tasks,models}.json
  ├── 0.1.0/{datasets,tasks,models}.json
  └── latest -> 0.1.0/    # stable URL for "current"
  

• The CLI optionally takes a --version flag that gets embedded as a _meta block in each manifest (so a fetched-in-isolation file is self-identifying).
• Website-side: loadEntities becomes version-aware (default: latest; explicit: ?v=0.0.15). That's a separate website-repo change.

Open sub-questions

• Should past versions ever be re-emitted (e.g., to backfill a new metadata field), or are they immutable once written?
• Version source: setuptools_scm-derived __version__, or curated semver tags only?
• Tip-of-main / pre-release representation — a "dev" alias under _web/entities/dev/, or omit entirely?
• For (a): adopting (b) might change which option is right. If collate_entities runs at release time off a tagged checkout that has been pip install-ed into the workflow's venv, the "installed vs source tree" distinction collapses — Option 2 (build on registries) becomes more attractive. Worth deciding (b)'s direction before locking in (a).

Acceptance criteria

• (a) Decide which unification option (status-quo + coherence test, build-on-registries, shared walker, push-into-registries). Implement and ship.
• (b) Workflow pivots from per-push regen to per-tag regen. _web/entities/ layout becomes versioned with index.json. CLI optionally embeds version metadata.
• Coordinate the matching website-side change so the leaderboard can read versioned entities (separate issue in the website repo).
• Migration plan for the existing _web/entities/*.json files (write a one-shot tagged version for the current state? Symlink? Keep the unversioned copy for backwards compat?).

Related

• PR #283 — CLI / module that this builds on.
• PR #285 — current "live" regen workflow this would change.
• Discussion: https://github.com/Medical-Event-Data-Standard/MEDS-DEV/pull/283#discussion_r3219530904

Start with the registry walkers in MEDS_DEV.datasets.init, MEDS_DEV.tasks.init, and MEDS_DEV.models.init, then compare them with MEDS_DEV.web.collate_entities and the regen workflow from PR #285. Review the linked design discussion before choosing an unification approach. Done means the selected walking change is shipped, tag-based versioned manifests and index.json work, migration is addressed, and the matching website change is coordinated.