THUDM / THUDM/slime

[Question] retool example: compute_log_probs(logits.clone(), tokens, tp_group) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.15 GiB.

Open
#1,951 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

Your Question

用https://github.com/THUDM/slime/tree/main/examples/retool 在4卡H200上进行训练，20~50步训练后出现

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.15 GiB. GPU 0 has a total capacity of 139.81 GiB of which 47.69 GiB is free. Process 22249 has 3.24 GiB memory in use. Including non-PyTorch memory, this process has 88.83 GiB memory in use. Of the allocated memory 82.55 GiB is allocated by PyTorch, and 2.06 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
                               ^^^^^^^^^^^^^^
    log_prob = compute_log_probs(logits.clone(), tokens, tp_group)
   File "/root/slime/slime/utils/ppo_utils.py", line 672, in calculate_log_probs_and_entropy
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
log_prob, entropy = calculate_log_probs_and_entropy(

运行参数和example提供的rl脚本完全一致。
不太明白设置了--max-tokens-per-gpu 9216之后，每一步的训练显存依然可能不一样吗？这种训练一定步数后OOM有什么可能的原因呢。

What I've Tried
  • 两次运行这个example都遇到一样的问题
Environment (if relevant)
  • slime version: docker slimerl/slime:v0.2.4
  • Python version: python3.12
  • PyTorch version:
  • CUDA/ROCm version:
  • GPU type and count: 4 x H200
  • OS:
Additional Context

第一次遇到这个报错的完整log：

(RolloutManager pid=1968846) [2026-05-25 09:11:40] rollout.py:1191 - perf 27: {'rollout/response_len/mean': 3398.0625, 'rollout/response_len/median': 2548.5, 'rollout/response_len/max': 10987, 'rollout/response_len/min': 409, 'rollout/zero_std/count_1.0': 4, 'rollout/repetition_frac': 0.0, 'rollout/truncated_ratio': 0.06640625, 'perf/rollout_time': 201.63432431221008, 'perf/tokens_per_gpu_per_sec': 1251.4077196960666, 'perf/longest_sample_tokens_per_sec': 514.6246818539139, 'perf/effective_tokens_per_gpu_per_sec': 1078.5663638461708, 'perf/longest_effective_sample_tokens_per_sec': 54.489730543038675}

(SGLangEngine pid=1974224) [2026-05-25 09:11:40 TP1] Cache flushed successfully!

(SGLangEngine pid=1974224) [2026-05-25 09:11:40 TP0] Cache flushed successfully!

(SGLangEngine pid=1974224) [2026-05-25 09:11:40 TP0] Decode batch, #running-req: 1, #token: 9690, token usage: 0.01, cuda graph: True, gen throughput (token/s): 297.24, #queue-req: 0 [repeated 56x across cluster]

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:41] timer.py:24 - Timer wake_up start

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:41] memory_utils.py:41 - [Rank 0] Memory-Usage before wake_up model: {'gpu': '0', 'total_GB': 139.81, 'free_GB': 133.34, 'used_GB': 6.47, 'allocated_GB': 23.22, 'reserved_GB': 23.62}

(MegatronTrainRayActor pid=2040576) [2026-05-25 09:11:46] reloadable_process_group.py:165 - Reloading 14 process groups in pid 2040576

(MegatronTrainRayActor pid=2040576) [2026-05-25 09:11:46] memory_utils.py:41 - [Rank 1] Memory-Usage after wake_up model: {'gpu': '1', 'total_GB': 139.81, 'free_GB': 110.47, 'used_GB': 29.34, 'allocated_GB': 23.22, 'reserved_GB': 23.62}

(SGLangEngine pid=1974234) [2026-05-25 09:11:40 TP1] Cache flushed successfully! [repeated 6x across cluster]

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:46] timer.py:32 - Timer wake_up end (elapsed: 5.2s)

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:46] timer.py:24 - Timer data_preprocess start

(MegatronTrainRayActor pid=2040577) [2026-05-25 09:11:41] memory_utils.py:41 - [Rank 2] Memory-Usage before wake_up model: {'gpu': '2', 'total_GB': 139.81, 'free_GB': 134.05, 'used_GB': 5.77, 'allocated_GB': 23.21, 'reserved_GB': 24.89} [repeated 3x across cluster]

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:46] timer.py:32 - Timer data_preprocess end (elapsed: 0.2s)

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:46] timer.py:32 - Timer train_wait end (elapsed: 221.5s)

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:46] timer.py:24 - Timer train start

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:11:46] timer.py:24 - Timer ref_log_probs start

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:12:24] timer.py:32 - Timer ref_log_probs end (elapsed: 37.4s)

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:12:24] timer.py:24 - Timer log_probs start

(MegatronTrainRayActor pid=2040577) [2026-05-25 09:11:46] reloadable_process_group.py:165 - Reloading 14 process groups in pid 2040577 [repeated 3x across cluster]

(MegatronTrainRayActor pid=2040577) [2026-05-25 09:11:46] memory_utils.py:41 - [Rank 2] Memory-Usage after wake_up model: {'gpu': '2', 'total_GB': 139.81, 'free_GB': 109.19, 'used_GB': 30.62, 'allocated_GB': 23.21, 'reserved_GB': 24.89} [repeated 3x across cluster]

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:12:37] timer.py:32 - Timer log_probs end (elapsed: 13.3s)

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:12:37] data.py:212 - rollout 27: {'rollout/response_lengths': 3942.60546875, 'rollout/rewards': -2.3836037144064903e-08, 'rollout/truncated': 0.06640625, 'rollout/rollout_log_probs': -0.17180030047893524, 'rollout/raw_reward': -0.009960937500000006, 'rollout/total_lengths': 4297.63671875, 'rollout/ref_log_probs': -0.17392803728580475, 'rollout/log_probs': -0.17246581614017487, 'rollout/advantages': -3.725290298461914e-08, 'rollout/returns': -3.725290298461914e-08}

(MegatronTrainRayActor pid=2025282) [2026-05-25 09:12:37] timer.py:24 - Timer actor_train start
Traceback (most recent call last):
  File "/root/slime/train.py", line 110, in <module>
    train(args)
  File "/root/slime/train.py", line 88, in train
    ray.get(actor_model.async_train(rollout_id, rollout_data_ref))
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2981, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1012, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::MegatronTrainRayActor.train() (pid=2025282, ip=10.252.197.29, actor_id=309d7b083d3cc636332c9bcb02000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x7fa62205de80>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/actor.py", line 376, in train
    return self.train_actor(rollout_id, rollout_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/actor.py", line 483, in train_actor
    train(
  File "/root/slime/slime/backends/megatron_utils/model.py", line 589, in train
    loss_dict, grad_norm = train_one_step(
                           ^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/model.py", line 420, in train_one_step
    losses_reduced = forward_backward_func(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 632, in forward_backward_no_pipelining
    output_tensor, num_tokens = forward_step(
                                ^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 422, in forward_step
    output_tensor, num_tokens = forward_step_calc_loss(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 248, in forward_step_calc_loss
    outputs = loss_func(output_tensor)
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/loss.py", line 999, in loss_function
    loss, log = func(args, batch, logits, sum_of_sample_mean)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/loss.py", line 650, in policy_loss_function
    _, log_probs_and_entropy = get_log_probs_and_entropy(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/loss.py", line 269, in get_log_probs_and_entropy
    log_prob, entropy = calculate_log_probs_and_entropy(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/utils/ppo_utils.py", line 672, in calculate_log_probs_and_entropy
    log_prob = compute_log_probs(logits.clone(), tokens, tp_group)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/utils/ppo_utils.py", line 158, in compute_log_probs
    return -fused_vocab_parallel_cross_entropy(logits, tokens, process_group)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/fusions/fused_cross_entropy.py", line 148, in fused_vocab_parallel_cross_entropy
    return _VocabParallelCrossEntropy.apply(vocab_parallel_logits, target, tp_group)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 581, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/fusions/fused_cross_entropy.py", line 104, in forward
    calculate_predicted_logits(
  File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/fusions/fused_cross_entropy.py", line 25, in calculate_predicted_logits
    @jit_fuser
  File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1130, in forward
    return compiled_fn(full_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353, in runtime_wrapper
    all_outs = call_func_at_runtime_with_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 129, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
                            ^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724, in inner_fn
    outs = compiled_fn(args)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526, in wrapper
    return compiled_fn(runtime_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 613, in __call__
    return self.current_callable(inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 3017, in run
    out = model(new_inputs)
          ^^^^^^^^^^^^^^^^^
  File "/tmp/torchinductor_root/kw/ckwznqkvnad34mx4nanvoeboe3klvqxwovwtfkcapcwmx47hvrdm.py", line 287, in call
    buf9 = empty_strided_cuda((s10, 1, 76032), (76032, 76032*s10, 1), torch.float32)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 29.39 GiB. GPU 0 has a total capacity of 139.81 GiB of which 19.41 GiB is free. Process 1990324 has 3.23 GiB memory in use. Including non-PyTorch memory, this process has 117.07 GiB memory in use. Of the allocated memory 92.06 GiB is allocated by PyTorch, and 20.79 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
  File "/root/slime/train.py", line 110, in <module>
    

train(args)
  File "/root/slime/train.py", line 88, in train
    

ray.get(actor_model.async_train(rollout_id, rollout_data_ref))
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2981, in get
values, debugger_breakpoint = worker.get_objects(
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1012, in get_objects
raise value.as_instanceof_cause()
ray.exceptions
RayTaskError(OutOfMemoryError)
: 
ray::MegatronTrainRayActor.train() (pid=2025282, ip=10.252.197.29, actor_id=309d7b083d3cc636332c9bcb02000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x7fa62205de80>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/actor.py", line 376, in train
    return self.train_actor(rollout_id, rollout_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/actor.py", line 483, in train_actor
    train(
  File "/root/slime/slime/backends/megatron_utils/model.py", line 589, in train
    loss_dict, grad_norm = train_one_step(
                           ^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/model.py", line 420, in train_one_step
    losses_reduced = forward_backward_func(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 632, in forward_backward_no_pipelining
    output_tensor, num_tokens = forward_step(
                                ^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 422, in forward_step
    output_tensor, num_tokens = forward_step_calc_loss(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 248, in forward_step_calc_loss
    outputs = loss_func(output_tensor)
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/loss.py", line 999, in loss_function
    loss, log = func(args, batch, logits, sum_of_sample_mean)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/loss.py", line 650, in policy_loss_function
    _, log_probs_and_entropy = get_log_probs_and_entropy(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/loss.py", line 269, in get_log_probs_and_entropy
    log_prob, entropy = calculate_log_probs_and_entropy(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/utils/ppo_utils.py", line 672, in calculate_log_probs_and_entropy
    log_prob = compute_log_probs(logits.clone(), tokens, tp_group)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/utils/ppo_utils.py", line 158, in compute_log_probs
    return -fused_vocab_parallel_cross_entropy(logits, tokens, process_group)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/fusions/fused_cross_entropy.py", line 148, in fused_vocab_parallel_cross_entropy
    return _VocabParallelCrossEntropy.apply(vocab_parallel_logits, target, tp_group)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 581, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/fusions/fused_cross_entropy.py", line 104, in forward
    calculate_predicted_logits(
  File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/fusions/fused_cross_entropy.py", line 25, in calculate_predicted_logits
    @jit_fuser
  File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1130, in forward
    return compiled_fn(full_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353, in runtime_wrapper
    all_outs = call_func_at_runtime_with_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 129, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
                            ^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724, in inner_fn
    outs = compiled_fn(args)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526, in wrapper
    return compiled_fn(runtime_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 613, in __call__
    return self.current_callable(inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 3017, in run
    out = model(new_inputs)
          ^^^^^^^^^^^^^^^^^
  File "/tmp/torchinductor_root/kw/ckwznqkvnad34mx4nanvoeboe3klvqxwovwtfkcapcwmx47hvrdm.py", line 287, in call
    buf9 = empty_strided_cuda((s10, 1, 76032), (76032, 76032*s10, 1), torch.float32)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 29.39 GiB. GPU 0 has a total capacity of 139.81 GiB of which 19.41 GiB is free. Process 1990324 has 3.23 GiB memory in use. Including non-PyTorch memory, this process has 117.07 GiB memory in use. Of the allocated memory 92.06 GiB is allocated by PyTorch, and 20.79 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Pre-submission Checklist

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the retool example with the reported parameters, then inspect slime/utils/ppo_utils.py at calculate_log_probs_and_entropy and compute_log_probs, followed by slime/backends/megatron_utils/loss.py. Compare per-step sequence lengths and GPU memory around rollout 27, especially the 58.15 GiB allocation; done means identifying the cause and confirming a reproducible fix or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.