Training freezes at 0% "train" after "gather" stage is complete
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.8k
- Forks
- 989
- Avg merge
- 6h 29m
- Merged PRs (30d)
- 85
Description
Running locally ART's 2048.ipynb notebook in Docker, skipping the first cell.
Training freezes at 0% "train" after the "gather" stage is complete. GPU utilization is at 0% in nvidia-smi.
Unsloth's Qwen3 GRPO notebook (without the use of ART) works as expected, training in it doesn't freeze.
NVIDIA RTX 5060 Ti
Dockerfile:
FROM quay.io/jupyter/pytorch-notebook:cuda12-python-3.12
USER root
RUN apt-get update && apt-get install -y build-essential
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN apt update && apt install -y cuda-toolkit
USER jovyan
RUN pip install openpipe-art==0.4.7 openpipe-art[backend]==0.4.7 --extra-index-url https://download.pytorch.org/whl/cu128 --extra-index-url https://wheels.vllm.ai/nightly
# Blackwell fix:
RUN pip uninstall -y xformers
RUN git clone --depth=1 https://github.com/facebookresearch/xformers --recursive && cd xformers && export TORCH_CUDA_ARCH_LIST="12.0" && python setup.py install
Output from the training cell:
gather: 100%
18/18 [01:35<00:00, 3.23s/it, reward=1.19, max_value=102, board_value=187, move_number=82.4, completion_tokens=21.8]
WARNING:weave.trace.op:Warning: Traces will not be logged. Call weave.init to log your traces to a project.
(subsequent messages of this type will be suppressed)
No "val/reward" metric found in history
Packed 18 trajectories into 15 sequences of length 6144
train: 0%
0/15 [00:00<?, ?it/s]
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\\ /| Num examples = 10,000,000 | Num Epochs = 3 | Total steps = 30,000,000
O^O/ \_/ \ Batch size per device = 2 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 1 | Total batch size (2 x 1 x 1) = 2
"-____-" Trainable parameters = 14,966,784 of 3,100,905,472 (0.48% trained)
Unsloth: Will smartly offload gradients to save VRAM!
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\\ /| Num examples = 10,000,000 | Num Epochs = 3 | Total steps = 60,000,000
O^O/ \_/ \ Batch size per device = 1 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 1 | Total batch size (1 x 1 x 1) = 1
"-____-" Trainable parameters = 14,966,784 of 3,100,905,472 (0.48% trained)
Any help is appreciated.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the freeze in the 2048.ipynb notebook using the provided Dockerfile and compare it with the working Unsloth Qwen3 GRPO notebook. Start at the transition from the completed gather stage to training, using the shown training output to identify where progress stops; done means training proceeds past 0% on the reported GPU setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, jupyter-notebook, python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100