OpenPipe / OpenPipe/ART

Training freezes at 0% "train" after "gather" stage is complete

Open
#346 20 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
10.8k
Forks
989
Avg merge
6h 29m
Merged PRs (30d)
85

Description

Running locally ART's 2048.ipynb notebook in Docker, skipping the first cell.
Training freezes at 0% "train" after the "gather" stage is complete. GPU utilization is at 0% in nvidia-smi.
Unsloth's Qwen3 GRPO notebook (without the use of ART) works as expected, training in it doesn't freeze.
NVIDIA RTX 5060 Ti
Dockerfile:

FROM quay.io/jupyter/pytorch-notebook:cuda12-python-3.12
USER root
RUN apt-get update && apt-get install -y build-essential
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN apt update && apt install -y cuda-toolkit
USER jovyan

RUN pip install openpipe-art==0.4.7 openpipe-art[backend]==0.4.7 --extra-index-url https://download.pytorch.org/whl/cu128 --extra-index-url https://wheels.vllm.ai/nightly

# Blackwell fix:
RUN pip uninstall -y xformers
RUN git clone --depth=1 https://github.com/facebookresearch/xformers --recursive && cd xformers && export TORCH_CUDA_ARCH_LIST="12.0" && python setup.py install

Output from the training cell:

gather: 100%
 18/18 [01:35<00:00,  3.23s/it, reward=1.19, max_value=102, board_value=187, move_number=82.4, completion_tokens=21.8]

WARNING:weave.trace.op:Warning: Traces will not be logged. Call weave.init to log your traces to a project.
 (subsequent messages of this type will be suppressed)

No "val/reward" metric found in history
Packed 18 trajectories into 15 sequences of length 6144

train:   0%
 0/15 [00:00<?, ?it/s]

==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1
   \\   /|    Num examples = 10,000,000 | Num Epochs = 3 | Total steps = 30,000,000
O^O/ \_/ \    Batch size per device = 2 | Gradient accumulation steps = 1
\        /    Data Parallel GPUs = 1 | Total batch size (2 x 1 x 1) = 2
 "-____-"     Trainable parameters = 14,966,784 of 3,100,905,472 (0.48% trained)

Unsloth: Will smartly offload gradients to save VRAM!

==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1
   \\   /|    Num examples = 10,000,000 | Num Epochs = 3 | Total steps = 60,000,000
O^O/ \_/ \    Batch size per device = 1 | Gradient accumulation steps = 1
\        /    Data Parallel GPUs = 1 | Total batch size (1 x 1 x 1) = 1
 "-____-"     Trainable parameters = 14,966,784 of 3,100,905,472 (0.48% trained)

Any help is appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the freeze in the 2048.ipynb notebook using the provided Dockerfile and compare it with the working Unsloth Qwen3 GRPO notebook. Start at the transition from the completed gather stage to training, using the shown training output to identify where progress stops; done means training proceeds past 0% on the reported GPU setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, jupyter-notebook, python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.