Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV

Expose ACES task criteria + dataset predicates to model `supervised` commands

Open
#314 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Models
Dominant language
Python
Stars
43
Forks
10
PR merge metrics
No merged PRs in 30d

Description

Problem

A model's supervised commands in model.yaml are formatted with a fixed set of template variables: dataset_dir, labels_dir, model_initialization_dir, output_dir, model_dir, split, demo (see model_commands / fmt_command in src/MEDS_DEV/models/__init__.py).

That set is sufficient for models that train a task-specific head and predict. It is not sufficient for zero-shot models that resolve generated trajectories against the task definition itself — those need the ACES task criteria file and the dataset predicates file:

  • TASKS[task]["criteria_fp"] — the ACES task config YAML
  • DATASETS[dataset]["predicates"] — the per-dataset predicates YAML

Both are already resolved internally by meds-dev-task, but they are never handed to model commands. A model command can't recover them either: it runs in an isolated venv that doesn't have MEDS_DEV installed.

Concrete blocker

This came up wiring MEDS-EIC-AR (PR #313). Its zero-shot supervised: predict needs to call meds-trajectory-evaluation's ZSACES_label, whose signature is:

ZSACES_label task.criteria_fp=<ACES task yaml> task.predicates_fp=<predicates yaml> \
    trajectories_dir=<...> output_dir=<...>

There is no template variable for task.criteria_fp or task.predicates_fp, so the command literally cannot be written today — str.format() would KeyError on an unknown placeholder.

Proposed change

Add two template variables, available to supervised commands:

  • {task_criteria_fp} — absolute path to the resolved ACES task criteria YAML.
  • {dataset_predicates_fp} — absolute path to the resolved per-dataset predicates YAML.

Both paths point at files inside the installed MEDS_DEV package on the shared filesystem, so a model command in an isolated venv can still read them. model_commands already has cfg.task_name / cfg.dataset_name available in full mode; the resolution is a registry lookup.

Open question worth deciding here: whether to also copy these into the labels_dir produced by meds-dev-task (so the task artifact is self-describing) instead of / in addition to passing them as template vars.

Related

  • Prerequisite for the zero-shot supervised: predict path in #313 (MEDS-EIC-AR).
  • Related to #304 (separating AR generation from inference) — that issue is about the lane structure; this one is the narrower "the model command can't even see the task definition" prerequisite.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/MEDS_DEV/models/init.py, examining model_commands and fmt_command in full mode, then trace how cfg.task_name and cfg.dataset_name resolve the ACES criteria and dataset predicates paths. Done means supervised commands can use absolute task_criteria_fp and dataset_predicates_fp values without a formatting KeyError; decide whether the resolved files also belong in the labels_dir artifact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.