Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV

CEHRBert evaluation issue

Open
#263 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43
Forks
10
PR merge metrics
No merged PRs in 30d

Description

MEDS evaluation command will run into an error; presumable because the predicted_boolean_value column is null:

(meds_dev_311) robin.vandewater@cx12:~/projects/MEDS_DEV_NEW$ meds-dev-evaluation predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="$EVALUATION_DIR/held_out"
[2025-10-18 11:46:05,023][MEDS_DEV.evaluation.__main__][INFO] - Running MEDS-Evaluation: meds-evaluation-cli predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out"
[2025-10-18 11:46:05,024][MEDS_DEV.utils][INFO] - Running command:
meds-evaluation-cli predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out"
Error executing job with overrides: ['predictions_path=/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/', 'output_dir=/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out']
Traceback (most recent call last):
  File "/sc/home/robin.vandewater/projects/MEDS_DEV_NEW/src/MEDS_DEV/evaluation/__main__.py", line 23, in main
    run_in_env(
  File "/sc/home/robin.vandewater/projects/MEDS_DEV_NEW/src/MEDS_DEV/utils.py", line 395, in run_in_env
    raise RuntimeError(
RuntimeError: Command failed with exit code 1:
SCRIPT:
meds-evaluation-cli predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out"
STDERR:
2025-10-18 11:46:07.416 | INFO     | meds_evaluation.__main__:main:20 - Starting evaluation with config:
predictions_path: /sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/
output_dir: /sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out
samples_per_subject: 4
resampling_seed: 0

2025-10-18 11:46:07.417 | INFO     | meds_evaluation.__main__:main:23 - Loading model predictions dataframe '/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/'
2025-10-18 11:46:07.444 | INFO     | meds_evaluation.__main__:main:31 - Running evaluation...
Error executing job with overrides: ['predictions_path=/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/', 'output_dir=/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out']
Traceback (most recent call last):
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_311/lib/python3.11/site-packages/meds_evaluation/__main__.py", line 32, in main
    result = evaluate_binary_classification(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_311/lib/python3.11/site-packages/meds_evaluation/evaluate.py", line 56, in evaluate_binary_classification
    validate_binary_classification_schema(predictions)
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_311/lib/python3.11/site-packages/meds_evaluation/schema.py", line 64, in validate_binary_classification_schema
    raise ValueError(
ValueError: Mismatched type for predicted_boolean_probability: expected Float32, got Float64

>>> pl.read_parquet("/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/")
shape: (798, 5)
┌────────────┬─────────────────────┬───────────────────────────────┬─────────────────────────┬───────────────┐
│ subject_id ┆ prediction_time     ┆ predicted_boolean_probability ┆ predicted_boolean_value ┆ boolean_value │
│ ---        ┆ ---                 ┆ ---                           ┆ ---                     ┆ ---           │
│ i64        ┆ datetime[ns]        ┆ f32                           ┆ null                    ┆ bool    

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command from the issue and inspect the parquet schema shown by pl.read_parquet. Start with src/MEDS_DEV/evaluation/main.py and the meds_evaluation evaluate_binary_classification/schema validation entry points from the traceback; compare the predicted_boolean_probability dtype with the validator expectation. Done means the evaluation command completes without the schema error and produces its output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.