For RLVMR: Question about SciWorld reward/success definition: done and score > 0

Hi, thanks for the great work!

I have a question about the reward and success definition in the SciWorld environment wrapper. This might be inherited from previous related work or an external implementation rather than being specific to this project, but I wanted to double-check whether this evaluation setup is intended.

The relevant code is in:

agent_system/environments/env_package/sciworld/envs.py

The reward is defined as:

def compute_reward(info, multi_modal=False):
    reward = 10.0 * float(info['won'])
    return reward

And in the worker step function:

observation, reward, done, info = env.step(action)

info['score'] = info.get('score', 0.0)
info['task_score'] = info['score']

isCompleted = done
prev_score = info['score']

info["won"] = isCompleted and info["score"] > 0
reward = compute_reward(info)

remote.send((observation, reward, isCompleted, info))

From my understanding, this means that the returned reward is:

reward = 10.0

whenever:

done == True and info["score"] > 0

Otherwise, the reward is 0.0.

My concern is that, in ScienceWorld, done=True may happen for different reasons. For example:

1. The agent truly completes the task, e.g. reaches the full score such as score = 100.
2. The episode terminates because it reaches envStepLimit.

In the second case, suppose the task has a maximum score of 100, but the agent only achieves a partial score before the step limit:

done = True
score = 10

Then the current code would set:

info["won"] = done and score > 0  # True
reward = 10.0

This seems to treat partial progress as a successful task completion, even though the agent may not have fully solved the task.

Could you clarify whether done and score > 0 is the intended criterion for task success in this benchmark? Or is it only intended to indicate that the episode ended with some positive progress?

I am wondering whether this evaluation is reasonable for measuring ScienceWorld task success. Would it be more appropriate to use a stricter success condition, such as reaching the maximum task score, e.g.:

info["won"] = done and info["score"] >= 100

or using an official ScienceWorld success signal if available?

Again, this may simply be a legacy design from prior work rather than an implementation choice of this repository. I just wanted to confirm the intended interpretation of reward = 10.0, and whether this evaluation is reasonable.

Thanks!

Start in agent_system/environments/env_package/sciworld/envs.py by tracing compute_reward and the worker step function. Verify how ScienceWorld distinguishes successful completion from envStepLimit termination, including any official success signal. Done means the intended success criterion is confirmed or corrected, with coverage for partial-score and time-limit cases.