Project-MONAI / Project-MONAI/MONAI
Insufficient error message when interrupted in `DataAnalyzer`
Open
Nobody has claimed this yet.
Feature request
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
Insufficient error message when interrupted in DataAnalyzer
root@5614474:/workspace/demo21/samm# git add .
72%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 4016/5587 [1:26:36<33:52, 1.29s/it]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1135, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/usr/lib/python3.10/queue.py", line 180, in get
self.not_empty.wait(remaining)
File "/usr/lib/python3.10/threading.py", line 324, in wait
gotit = waiter.acquire(True, timeout)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3683) is killed by signal: Killed.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/workspace/data/analyzer.py", line 203, in compute_label_histogram_stats
analyzer.get_all_case_stats(transform_list=get_label_stats_transforms())
File "/opt/monai/monai/apps/auto3dseg/data_analyzer.py", line 230, in get_all_case_stats
result_bycase = self._get_all_case_stats(0, 1, None, key, transform_list)
File "/opt/monai/monai/apps/auto3dseg/data_analyzer.py", line 333, in _get_all_case_stats
for batch_data in tqdm(dataloader) if (has_tqdm and rank == 0) else dataloader:
File "/usr/local/lib/python3.10/dist-packages/tqdm/std.py", line 1178, in __iter__
for obj in iterable:
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 633, in __next__
data = self._next_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1331, in _next_data
idx, data = self._get_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1287, in _get_data
success, data = self._try_get_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1148, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 3683) exited unexpectedly
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with monai/apps/auto3dseg/data_analyzer.py, especially get_all_case_stats and _get_all_case_stats, and review the reported call from /workspace/data/analyzer.py in compute_label_histogram_stats. Reproduce the DataAnalyzer workflow that interrupts with a killed DataLoader worker; done means the resulting error message provides more useful information than the current worker-exited traceback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100