Project-MONAI / Project-MONAI/tutorials

can't run on large datasets

Open
#1,759 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
2.5k
Forks
803
Avg merge
6d 22h
Merged PRs (30d)
3

Description

When I correctly follow the steps in the auto3dseg_hello_world.ipynb notebook, set the corresponding paths and parameters, and run it in an environment with 48G of GPU memory, I encounter the error RuntimeError: Pin memory thread exited unexpectedly while attempting to train on a dataset with 300 .nii.gz images. In contrast, when using a dataset with 20 images, the training proceeds smoothly under the exact same conditions. During the training process with the 300-image dataset, I monitored the GPU memory usage and found it to be less than 70%. However, the error keeps occurring inexplicably. Could there be an issue with the get_data step?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running auto3dseg_hello_world.ipynb with the reported paths and parameters, comparing the 20-image and 300-image datasets on a 48G GPU environment. Inspect the get_data step and the point where training raises “Pin memory thread exited unexpectedly.” Done means the notebook can train on the larger dataset without this error, with the smaller-dataset behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.