THUDM / THUDM/slime

UX/Unintended bug: sample.reward is None on aborted sample

Open Beginner friendly
#200 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

Hi @zhuzilin, I think I found a subtle unintended UX issue. If a sample is aborted, sample.reward will be None because that is the default. This can easily be caused by using a --custom-generate-function-path where you might want to abort certain samples like in Search-R1 if a tool call fails.

However, aborting a sample is not safe as it will trigger an error when converting the list of rewards to a PyTorch Tensor in GRPO:

https://github.com/THUDM/slime/blob/3bbcbcc704d5f067ceaf6fb84366ffb1ca41defa/slime/backends/utils/data.py#L78-L88

I suggest a minor patch like below or another way to handle None reward values (pending further testing if this works).

diff --git a/slime/rollout/sglang_rollout.py b/slime/rollout/sglang_rollout.py
index 51150f3..5b46358 100644
--- a/slime/rollout/sglang_rollout.py
+++ b/slime/rollout/sglang_rollout.py
@@ -139,6 +139,7 @@ async def generate_and_rm(args, sample: Sample, sampling_params: dict, evaluatio
     # generate
     async with state.semaphore:
         if state.aborted:
+            sample.reward = 0.0
             sample.status = Sample.Status.ABORTED
             return sample

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in slime/rollout/sglang_rollout.py at generate_and_rm, then inspect slime/backends/utils/data.py around lines 78-88 where rewards are converted to a PyTorch tensor. Verify the aborted-sample path produces a tensor-compatible reward and that GRPO no longer errors when a sample is aborted, using the relevant existing test or reproduction path if available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.