NVIDIA / NVIDIA/Megatron-LM

[BUG]DataLoader worker is killed by signal, memory insufficient

Open
#1,657 0 comments 0 reactions 0 assignees View on GitHub
bug module: data pipeline
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

when sequence length is bigger than 90K,I get dataloader worker killed signal, and max memory usage reach 100%(about 1000Gb). The num-workers is set to 0, batch size is set to 1. But when sequence length is 64K, trainning succeed with the same config. Anyone know how to solve the problem?
error details::
training ...
Setting rerun_state_machine.current_iteration to 0...
[before the start of training step] datetime: 2025-06-30 01:59:29
[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1130, in _try_get_data
[rank0]: data = self._data_queue.get(timeout=timeout)
[rank0]: File "/usr/lib/python3.10/queue.py", line 180, in get
[rank0]: self.not_empty.wait(remaining)
[rank0]: File "/usr/lib/python3.10/threading.py", line 324, in wait
[rank0]: gotit = waiter.acquire(True, timeout)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
[rank0]: _error_if_any_worker_fails()
[rank0]: RuntimeError: DataLoader worker (pid 2637) is killed by signal: Killed.

[rank0]: The above exception was the direct cause of the following exception:

Contributor guide

Open the contributing guide

Research direction

Start with the PyTorch paths shown in the traceback: torch/utils/data/dataloader.py and torch/utils/data/_utils/signal_handling.py. Reproduce the same configuration at sequence lengths of 64K and over 90K, then trace the memory growth and worker termination. Done means the cause is identified and the behavior is verified with an appropriate regression check or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data-engineering, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.