[BUG]DataLoader worker is killed by signal, memory insufficient
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
when sequence length is bigger than 90K,I get dataloader worker killed signal, and max memory usage reach 100%(about 1000Gb). The num-workers is set to 0, batch size is set to 1. But when sequence length is 64K, trainning succeed with the same config. Anyone know how to solve the problem?
error details::
training ...
Setting rerun_state_machine.current_iteration to 0...
[before the start of training step] datetime: 2025-06-30 01:59:29
[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1130, in _try_get_data
[rank0]: data = self._data_queue.get(timeout=timeout)
[rank0]: File "/usr/lib/python3.10/queue.py", line 180, in get
[rank0]: self.not_empty.wait(remaining)
[rank0]: File "/usr/lib/python3.10/threading.py", line 324, in wait
[rank0]: gotit = waiter.acquire(True, timeout)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
[rank0]: _error_if_any_worker_fails()
[rank0]: RuntimeError: DataLoader worker (pid 2637) is killed by signal: Killed.
[rank0]: The above exception was the direct cause of the following exception:
Contributor guide
Research direction
Start with the PyTorch paths shown in the traceback: torch/utils/data/dataloader.py and torch/utils/data/_utils/signal_handling.py. Reproduce the same configuration at sequence lengths of 64K and over 90K, then trace the memory growth and worker termination. Done means the cause is identified and the behavior is verified with an appropriate regression check or documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100