Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV

Document the website integration / results pipeline

Open
#282 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation priority:medium Website / Branding
Dominant language
Python
Stars
43
Forks
10
PR merge metrics
No merged PRs in 30d

Description

Background

The end-to-end pipeline that takes a benchmark result from a contributor and surfaces it on the MEDS website involves several moving pieces:

  • An issue template (.github/ISSUE_TEMPLATE/benchmark-result.yml) drives result submission
  • upload_benchmark_result.yaml validates the JSON, commits to _results branch, closes the issue
  • aggregate_benchmark_results.yaml runs _web/scripts/aggregate_results.py on push to _results
  • _web/entities/*.json (manually maintained, see #186/#187/#188) provides the dataset/task/model catalog
  • The website fetches _web/results/all_results.json and _web/entities/*.json at runtime

This is documented nowhere outside individual workflow YAML files. New contributors and maintainers can't trace what happens after an issue is filed, and several integration points have already broken silently (#238).

What needs to be done

Add docs (docs/website-integration.md or similar) covering:

  1. Architecture diagram: producers (CI workflows, issue handlers) → consumers (_results / _web orphan branches) → website fetch path.
  2. Branches:
    • main — code, tasks, datasets, models
    • _results — orphan branch with _results/<issue#>/result.json per benchmark submission
    • _web — orphan branch with entities/*.json (manifests) and results/all_results.json (aggregated leaderboard)
  3. Workflows and what triggers each, including known limitations (e.g., GITHUB_TOKEN not triggering downstream workflows — see #238).
  4. Manual interventions: when and how to run workflow_dispatch (currently the only working way to refresh _web/results/all_results.json until #238 is fixed).
  5. Schema docs: shape of result.json, of the entities files, and how the website consumes them. Link to website-side parsers (src/lib/MEDS-DEV/{load,entities,parse_tree}.ts).
  6. Adding a new dataset/task/model: the contributor checklist, including what to expect on the website afterward.

Acceptance criteria

  • A new contributor can read the doc and understand "I labeled my issue, why isn't it on the website?" without grepping workflows.
  • A maintainer can read the doc and know how to manually unblock the pipeline if a workflow fails.

Related

  • Cross-cuts #238 (auto-aggregation), #186/#187/#188 (entity manifests), #280/#281 (proposed automation work).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading .github/ISSUE_TEMPLATE/benchmark-result.yml, upload_benchmark_result.yaml, aggregate_benchmark_results.yaml, and _web/scripts/aggregate_results.py to trace submission through the _results and _web branches. Then inspect the website parsers in src/lib/MEDS-DEV/{load,entities,parse_tree}.ts and the referenced entity and result files. Done means docs explain triggers, schemas, manual workflow_dispatch recovery, contributor steps, and how to diagnose a missing website result.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python, typescript
Domain
ci-cd, documentation, web-dev
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.