Medical-Event-Data-Standard/MEDS-DEV

View on GitHub

The MEDS Decentralized Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health

Stars: 43
Forks: 10
Open beginner issues: 0
Indexed issues: 77

Dominant language: Python
License: MIT
Last GitHub push: Apr 16, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: help wanted good first issue
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

77 open issues indexed

Assess PATH/environment semantics of command execution (venv exposure, bare-name resolution, -m runnability) Open

Difficulty 5/5 Over a week Newbie friendliness 42/100

Medical-Event-Data-Standard/MEDS-DEV#323 · Opened Jul 27, 2026
Native Windows dataset builds fail because virtualenv and command execution assume POSIX paths Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

Medical-Event-Data-Standard/MEDS-DEV#316 · Opened Jul 15, 2026
MIMIC-IV `predicates.yaml` lab codes are `LAB//<itemid>//<unit>`, but the pinned ETL now emits `LAB//RESULT//<itemid>//<unit>` Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

Medical-Event-Data-Standard/MEDS-DEV#315 · 1 comment · Opened Jul 13, 2026
Expose ACES task criteria + dataset predicates to model `supervised` commands Open

enhancement Models

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Medical-Event-Data-Standard/MEDS-DEV#314 · Opened May 14, 2026
Separate autoregressive generation from downstream inference for AR zero-shot models Open

enhancement Models

Difficulty 5/5 Over a week Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#304 · Opened May 13, 2026
Add larger MEDS-Tab variants alongside meds_tab/tiny Open

enhancement MEDS-Tab Models

Difficulty 4/5 3-5 days Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#303 · Opened May 13, 2026
Add MEDS-EIC-AR as a registered model Open

enhancement Models

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Medical-Event-Data-Standard/MEDS-DEV#302 · Opened May 13, 2026
Add meds-dev-submit-result CLI: client-side submission via gh Open

enhancement priority:medium Usability/Interface Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Medical-Event-Data-Standard/MEDS-DEV#297 · Opened May 11, 2026
Make result-submission form fields meaningful (dropdowns + registry-driven regen) Open

Code Cleanliness/Tech Debt priority:medium Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 50/100

Medical-Event-Data-Standard/MEDS-DEV#296 · Opened May 11, 2026
act-based workflow smoke tests + sandbox infrastructure Open

priority:medium Testing Website / Branding

Difficulty 5/5 Over a week Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#293 · Opened May 11, 2026
Add concrete schemas for result blobs and entity manifests Open

Code Cleanliness/Tech Debt priority:medium Results storage and tracking Website / Branding

Difficulty 5/5 Over a week Newbie friendliness 25/100

Medical-Event-Data-Standard/MEDS-DEV#290 · Opened May 11, 2026
Unify entity-walking code paths; make entity collation version-aware Open

Code Cleanliness/Tech Debt priority:medium Results Interpretation Website / Branding

Difficulty 5/5 Over a week Newbie friendliness 25/100

Medical-Event-Data-Standard/MEDS-DEV#289 · Opened May 11, 2026
Document the website integration / results pipeline Open

documentation priority:medium Website / Branding

Difficulty 3/5 1-2 days Newbie friendliness 68/100

Medical-Event-Data-Standard/MEDS-DEV#282 · Opened May 8, 2026
Add workflow to auto-regenerate _web entity manifests on source changes Open

priority:high Testing Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 68/100

Medical-Event-Data-Standard/MEDS-DEV#281 · Opened May 8, 2026
Promote web tooling (collate_entities, aggregate_results) into the package Open

Code Cleanliness/Tech Debt priority:high Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Medical-Event-Data-Standard/MEDS-DEV#280 · Opened May 8, 2026
Seperate data preprocessing from training/inference Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#270 · Opened Nov 26, 2025
overwrite=True: runs into directory not empty errors Open

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

Medical-Event-Data-Standard/MEDS-DEV#268 · Opened Nov 18, 2025
OSError: [Errno 122] Disk quota exceeded Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#266 · 3 comments · Opened Oct 26, 2025
Unit conversion with the existing tasks Open

Datasets Tasks

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Medical-Event-Data-Standard/MEDS-DEV#265 · 3 comments · Opened Oct 23, 2025
Add FEMR tasks/labelers Open

enhancement help wanted

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#264 · Opened Oct 18, 2025
CEHRBert evaluation issue Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#263 · 1 comment · Opened Oct 18, 2025
meds-dev-evaluation documentation clarity Open

Difficulty 1/5 Under an hour Newbie friendliness 68/100

Medical-Event-Data-Standard/MEDS-DEV#253 · 1 reaction · Opened Oct 13, 2025
Tests badge points to wrong workflow file. Open

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#252 · Opened Oct 13, 2025
MEDS-TAB is causing test failures Open

Medical-Event-Data-Standard/MEDS-DEV#251 · 1 assignee · Opened Oct 13, 2025
[Model] Add the model CEHR-XGPT Open

Models

Medical-Event-Data-Standard/MEDS-DEV#247 · 1 assignee · Opened Sep 29, 2025
Track computational resource usage during model run Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

Medical-Event-Data-Standard/MEDS-DEV#246 · Opened Sep 26, 2025
Development documentation Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

Medical-Event-Data-Standard/MEDS-DEV#245 · 4 comments · Opened Sep 25, 2025
Should we rename "Models" "Algorithms"? Or "Model Training Recipes"? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

Medical-Event-Data-Standard/MEDS-DEV#244 · 5 comments · Opened Sep 24, 2025
Can't run in colab Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

Medical-Event-Data-Standard/MEDS-DEV#242 · Opened Jul 24, 2025
GenHPF crash Open

bug GenHPF priority:high

Medical-Event-Data-Standard/MEDS-DEV#241 · 9 comments · 2 assignees · Opened Jul 1, 2025
NaNs are not valid json and need to be parsed out into nulls in the result JSON blurbs. Open

priority:high Results storage and tracking Website / Branding

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#239 · 1 comment · Opened May 30, 2025
aggregate benchmark results workflow is not triggering on new result submissions Open

bug priority:high Website / Branding

Difficulty 3/5 1-2 days Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#238 · 1 comment · Opened May 30, 2025
Abnormal Lab Tasks permit 100% accuracy Open

priority:high Tasks

Medical-Event-Data-Standard/MEDS-DEV#237 · 15 comments · 1 reaction · 3 assignees · Opened May 30, 2025
Committing results too quickly via issues results in action failures Open

priority:medium Results storage and tracking

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

Medical-Event-Data-Standard/MEDS-DEV#236 · 1 comment · Opened May 30, 2025
CEHRBERT Error Open

CEHR-BERT priority:medium

Medical-Event-Data-Standard/MEDS-DEV#213 · 1 assignee · Opened May 30, 2025
Update trilom change file detector to https://github.com/tj-actions/changed-files in PR code quality checker. Open

Code Cleanliness/Tech Debt priority:low

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#212 · 1 comment · Opened May 20, 2025
Upgrade to MEDS v0.4 Open

Code Cleanliness/Tech Debt priority:high Usability/Interface

Difficulty 5/5 Over a week Newbie friendliness 10/100

Medical-Event-Data-Standard/MEDS-DEV#208 · Opened May 1, 2025
MEDS-DEV Results should track dataset metadata (e.g., versions) Open

enhancement good first issue priority:high Results storage and tracking

Difficulty 5/5 Over a week Newbie friendliness 25/100

Medical-Event-Data-Standard/MEDS-DEV#206 · Opened Mar 26, 2025
[Model] Add the model CORE-BEHRT Open

BLOCKED enhancement help wanted Models priority:medium

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Medical-Event-Data-Standard/MEDS-DEV#205 · 1 comment · Opened Mar 21, 2025
We should support evaluation-only mode for pre-trained models distributed with weights Open

enhancement Experimental API help wanted Models priority:medium Usability/Interface

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#204 · 1 reaction · Opened Mar 14, 2025
[Dataset] Adds dataset EHR-Shot Open

Datasets help wanted priority:medium

Medical-Event-Data-Standard/MEDS-DEV#203 · 1 reaction · 2 assignees · Opened Mar 14, 2025
Evaluation should store output in a file, not a directory, and hydra logging should simply be disabled. Open

BLOCKED Code Cleanliness/Tech Debt enhancement priority:low

Difficulty 3/5 1-2 days Newbie friendliness 38/100

Medical-Event-Data-Standard/MEDS-DEV#191 · Opened Mar 12, 2025
We need to export a code-endpoint releasing the information about supported models in a friendly format. Open

Blocking External Use Models priority:critical Testing Usability/Interface Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Medical-Event-Data-Standard/MEDS-DEV#188 · 1 comment · Opened Mar 11, 2025
We need to export a code-endpoint releasing the information about supported tasks in a friendly format. Open

Blocking External Use priority:critical Tasks Testing Usability/Interface Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#187 · 1 comment · Opened Mar 11, 2025
We need to export a code-endpoint releasing the information about supported datasets in a friendly format. Open

Blocking External Use Datasets priority:critical Testing Usability/Interface Website / Branding

Difficulty 4/5 3-5 days Newbie friendliness 25/100

Medical-Event-Data-Standard/MEDS-DEV#186 · 1 comment · Opened Mar 11, 2025
Further improvements to `abnormal_lab` tasks. Open

enhancement help wanted priority:critical Tasks

Medical-Event-Data-Standard/MEDS-DEV#185 · 7 comments · 5 assignees · Opened Mar 11, 2025
We should add some ED related tasks to leverage MIMIC-IV's ED data. Open

enhancement priority:high Tasks

Medical-Event-Data-Standard/MEDS-DEV#184 · 1 assignee · Opened Mar 11, 2025
We should incorporate the task definition rubric Open

priority:low Tasks Usability/Interface

Medical-Event-Data-Standard/MEDS-DEV#182 · 1 assignee · Opened Mar 11, 2025
Additional ICU Mortality related tasks should be added. Open

priority:high Tasks

Medical-Event-Data-Standard/MEDS-DEV#177 · 3 comments · 1 assignee · Opened Mar 10, 2025
Test code should use more plugins and built-in pytest fixtures for code simplicity and performance. Open

Code Cleanliness/Tech Debt Computational Efficiency enhancement priority:low Testing

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Medical-Event-Data-Standard/MEDS-DEV#160 · Opened Mar 5, 2025

Medical-Event-Data-Standard/MEDS-DEV

Get new issues in your inbox