Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV
Expose ACES task criteria + dataset predicates to model `supervised` commands
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Problem
A model's supervised commands in model.yaml are formatted with a fixed set of template variables: dataset_dir, labels_dir, model_initialization_dir, output_dir, model_dir, split, demo (see model_commands / fmt_command in src/MEDS_DEV/models/__init__.py).
That set is sufficient for models that train a task-specific head and predict. It is not sufficient for zero-shot models that resolve generated trajectories against the task definition itself — those need the ACES task criteria file and the dataset predicates file:
TASKS[task]["criteria_fp"]— the ACES task config YAMLDATASETS[dataset]["predicates"]— the per-dataset predicates YAML
Both are already resolved internally by meds-dev-task, but they are never handed to model commands. A model command can't recover them either: it runs in an isolated venv that doesn't have MEDS_DEV installed.
Concrete blocker
This came up wiring MEDS-EIC-AR (PR #313). Its zero-shot supervised: predict needs to call meds-trajectory-evaluation's ZSACES_label, whose signature is:
ZSACES_label task.criteria_fp=<ACES task yaml> task.predicates_fp=<predicates yaml> \
trajectories_dir=<...> output_dir=<...>
There is no template variable for task.criteria_fp or task.predicates_fp, so the command literally cannot be written today — str.format() would KeyError on an unknown placeholder.
Proposed change
Add two template variables, available to supervised commands:
{task_criteria_fp}— absolute path to the resolved ACES task criteria YAML.{dataset_predicates_fp}— absolute path to the resolved per-dataset predicates YAML.
Both paths point at files inside the installed MEDS_DEV package on the shared filesystem, so a model command in an isolated venv can still read them. model_commands already has cfg.task_name / cfg.dataset_name available in full mode; the resolution is a registry lookup.
Open question worth deciding here: whether to also copy these into the labels_dir produced by meds-dev-task (so the task artifact is self-describing) instead of / in addition to passing them as template vars.
Related
- Prerequisite for the zero-shot
supervised: predictpath in #313 (MEDS-EIC-AR). - Related to #304 (separating AR generation from inference) — that issue is about the lane structure; this one is the narrower "the model command can't even see the task definition" prerequisite.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/MEDS_DEV/models/init.py, examining model_commands and fmt_command in full mode, then trace how cfg.task_name and cfg.dataset_name resolve the ACES criteria and dataset predicates paths. Done means supervised commands can use absolute task_criteria_fp and dataset_predicates_fp values without a formatting KeyError; decide whether the resolved files also belong in the labels_dir artifact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100