Medical-Event-Data-Standard / Medical-Event-Data-Standard/MEDS-DEV
GenHPF crash
Open
@Jwoo5 is already working on this.
Since Jul 4, 2025.
bug
GenHPF
priority:high
- Dominant language
- Python
- Stars
- 43
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am running GenHPF with a CPU node with 8 cores and 10GB of memory per core. I am getting the error below. Perhaps allocating more resources would help? What would be the recommended amount and should this be run on a GPU anyways?
Perhaps most suitable to check would be @hoon9405 ? Thank you!
STDERR:
/sc/arion/work/vander09/conda/envs/meds_dev/lib/python3.12/multiprocessing/popen_fork.py:66: RuntimeWarning: Using fork() can cause Polars to deadlock in the child proc
ess.
In addition, using fork() with Python in general is a recipe for mysterious
deadlocks and crashes.
The most likely reason you are seeing this error is because you are using the
multiprocessing module on Linux, which uses fork() by default. This will be
fixed in Python 3.14. Until then, you want to use the "spawn" context instead.
See https://docs.pola.rs/user-guide/misc/multiprocessing/ for details.
If you really know what your doing, you can silence this warning with the warning module
or by setting POLARS_ALLOW_FORKING_THREAD=1.
self.pid = os.fork()
Error executing job with overrides: ['dataset.data=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/
genhpf/abnormal_lab/general/general_hypertension/data', 'checkpoint.save_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25
_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hypertension/checkpoints', 'common.debug=False']
Traceback (most recent call last):
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/scripts/train.py", line 349, in hydra_main
distributed_utils.call_main(cfg, main)
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/utils/distributed_utils.py", line 203, in call_main
main(cfg, **kwargs)
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/scripts/train.py", line 114, in main
valid_losses, should_stop = train(cfg, trainer, datasets, i)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/work/vander09/conda/envs/meds_dev/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/scripts/train.py", line 185, in train
log_output = trainer.train_step(sample)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/work/vander09/conda/envs/meds_dev/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/trainer.py", line 243, in train_step
loss, sample_size, logging_output = self.criterion(self.model, sample)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/criterions/criterion.py", line 60, in forward net_output = model(**sample["net_input"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hypertension/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/models/genhpf_predictor.py", line 56, in forward x, padding_mask = super().forward(
^^^^^^^^^^^^^^^^ File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/genhpf/models/genhpf.py", line 207, in forward
x += self.digit_place_embeddings(dpe_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hyperte
nsion/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hypertension/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hypertension/.venv/lib/python3.12/site-packages/torch/nn/modules/sparse.py", line 190, in forward
return F.embedding(
^^^^^^^^^^^^
File "/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/full_omop_25_04_29/MEDS_cohort/models/genhpf/abnormal_lab/general/general_hypertension/.venv/lib/python3.12/site-packages/torch/nn/functional.py", line 2551, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index out of range in self
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.