Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV
CEHRBert evaluation issue
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
MEDS evaluation command will run into an error; presumable because the predicted_boolean_value column is null:
(meds_dev_311) robin.vandewater@cx12:~/projects/MEDS_DEV_NEW$ meds-dev-evaluation predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="$EVALUATION_DIR/held_out"
[2025-10-18 11:46:05,023][MEDS_DEV.evaluation.__main__][INFO] - Running MEDS-Evaluation: meds-evaluation-cli predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out"
[2025-10-18 11:46:05,024][MEDS_DEV.utils][INFO] - Running command:
meds-evaluation-cli predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out"
Error executing job with overrides: ['predictions_path=/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/', 'output_dir=/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out']
Traceback (most recent call last):
File "/sc/home/robin.vandewater/projects/MEDS_DEV_NEW/src/MEDS_DEV/evaluation/__main__.py", line 23, in main
run_in_env(
File "/sc/home/robin.vandewater/projects/MEDS_DEV_NEW/src/MEDS_DEV/utils.py", line 395, in run_in_env
raise RuntimeError(
RuntimeError: Command failed with exit code 1:
SCRIPT:
meds-evaluation-cli predictions_path="/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/" output_dir="/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out"
STDERR:
2025-10-18 11:46:07.416 | INFO | meds_evaluation.__main__:main:20 - Starting evaluation with config:
predictions_path: /sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/
output_dir: /sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out
samples_per_subject: 4
resampling_seed: 0
2025-10-18 11:46:07.417 | INFO | meds_evaluation.__main__:main:23 - Loading model predictions dataframe '/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/'
2025-10-18 11:46:07.444 | INFO | meds_evaluation.__main__:main:31 - Running evaluation...
Error executing job with overrides: ['predictions_path=/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/', 'output_dir=/sc/home/robin.vandewater/datasets/meds/eICU/results/mortality/in_icu/first_24h/cehrbert/held_out']
Traceback (most recent call last):
File "/sc/home/robin.vandewater/conda3/envs/meds_dev_311/lib/python3.11/site-packages/meds_evaluation/__main__.py", line 32, in main
result = evaluate_binary_classification(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/home/robin.vandewater/conda3/envs/meds_dev_311/lib/python3.11/site-packages/meds_evaluation/evaluate.py", line 56, in evaluate_binary_classification
validate_binary_classification_schema(predictions)
File "/sc/home/robin.vandewater/conda3/envs/meds_dev_311/lib/python3.11/site-packages/meds_evaluation/schema.py", line 64, in validate_binary_classification_schema
raise ValueError(
ValueError: Mismatched type for predicted_boolean_probability: expected Float32, got Float64
>>> pl.read_parquet("/sc/home/robin.vandewater/datasets/meds/AUMCdb/models/mortality/in_icu/first_24h/cehrbert/first_24h/test_predictions/")
shape: (798, 5)
┌────────────┬─────────────────────┬───────────────────────────────┬─────────────────────────┬───────────────┐
│ subject_id ┆ prediction_time ┆ predicted_boolean_probability ┆ predicted_boolean_value ┆ boolean_value │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ datetime[ns] ┆ f32 ┆ null ┆ bool
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the command from the issue and inspect the parquet schema shown by pl.read_parquet. Start with src/MEDS_DEV/evaluation/main.py and the meds_evaluation evaluate_binary_classification/schema validation entry points from the traceback; compare the predicted_boolean_probability dtype with the validator expectation. Done means the evaluation command completes without the schema error and produces its output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100