Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV
Document the website integration / results pipeline
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Background
The end-to-end pipeline that takes a benchmark result from a contributor and surfaces it on the MEDS website involves several moving pieces:
- An issue template (
.github/ISSUE_TEMPLATE/benchmark-result.yml) drives result submission upload_benchmark_result.yamlvalidates the JSON, commits to_resultsbranch, closes the issueaggregate_benchmark_results.yamlruns_web/scripts/aggregate_results.pyon push to_results_web/entities/*.json(manually maintained, see #186/#187/#188) provides the dataset/task/model catalog- The website fetches
_web/results/all_results.jsonand_web/entities/*.jsonat runtime
This is documented nowhere outside individual workflow YAML files. New contributors and maintainers can't trace what happens after an issue is filed, and several integration points have already broken silently (#238).
What needs to be done
Add docs (docs/website-integration.md or similar) covering:
- Architecture diagram: producers (CI workflows, issue handlers) → consumers (
_results/_weborphan branches) → website fetch path. - Branches:
main— code, tasks, datasets, models_results— orphan branch with_results/<issue#>/result.jsonper benchmark submission_web— orphan branch withentities/*.json(manifests) andresults/all_results.json(aggregated leaderboard)
- Workflows and what triggers each, including known limitations (e.g.,
GITHUB_TOKENnot triggering downstream workflows — see #238). - Manual interventions: when and how to run
workflow_dispatch(currently the only working way to refresh_web/results/all_results.jsonuntil #238 is fixed). - Schema docs: shape of
result.json, of the entities files, and how the website consumes them. Link to website-side parsers (src/lib/MEDS-DEV/{load,entities,parse_tree}.ts). - Adding a new dataset/task/model: the contributor checklist, including what to expect on the website afterward.
Acceptance criteria
- A new contributor can read the doc and understand "I labeled my issue, why isn't it on the website?" without grepping workflows.
- A maintainer can read the doc and know how to manually unblock the pipeline if a workflow fails.
Related
- Cross-cuts #238 (auto-aggregation), #186/#187/#188 (entity manifests), #280/#281 (proposed automation work).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading .github/ISSUE_TEMPLATE/benchmark-result.yml, upload_benchmark_result.yaml, aggregate_benchmark_results.yaml, and _web/scripts/aggregate_results.py to trace submission through the _results and _web branches. Then inspect the website parsers in src/lib/MEDS-DEV/{load,entities,parse_tree}.ts and the referenced entity and result files. Done means docs explain triggers, schemas, manual workflow_dispatch recovery, contributor steps, and how to diagnose a missing website result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python, typescript
- Domain
- ci-cd, documentation, web-dev
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100