Project-MONAI / Project-MONAI/MONAI

Insufficient error message when interrupted in `DataAnalyzer`

Open
#7,327 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature request
Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Insufficient error message when interrupted in DataAnalyzer

root@5614474:/workspace/demo21/samm# git add .
 72%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                           | 4016/5587 [1:26:36<33:52,  1.29s/it]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1135, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.10/queue.py", line 180, in get
    self.not_empty.wait(remaining)
  File "/usr/lib/python3.10/threading.py", line 324, in wait
    gotit = waiter.acquire(True, timeout)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3683) is killed by signal: Killed. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/data/analyzer.py", line 203, in compute_label_histogram_stats
    analyzer.get_all_case_stats(transform_list=get_label_stats_transforms())
  File "/opt/monai/monai/apps/auto3dseg/data_analyzer.py", line 230, in get_all_case_stats
    result_bycase = self._get_all_case_stats(0, 1, None, key, transform_list)
  File "/opt/monai/monai/apps/auto3dseg/data_analyzer.py", line 333, in _get_all_case_stats
    for batch_data in tqdm(dataloader) if (has_tqdm and rank == 0) else dataloader:
  File "/usr/local/lib/python3.10/dist-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1331, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1287, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1148, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 3683) exited unexpectedly

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with monai/apps/auto3dseg/data_analyzer.py, especially get_all_case_stats and _get_all_case_stats, and review the reported call from /workspace/data/analyzer.py in compute_label_histogram_stats. Reproduce the DataAnalyzer workflow that interrupts with a killed DataLoader worker; done means the resulting error message provides more useful information than the current worker-exited traceback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.