huggingface / huggingface/trl

[harbor/openreward] Adopt environment-owned reward (#6238) and dataset (#6349)

Open
#6,354 2 comments 0 reactions 1 assignee Claimed by @adithya-s-k View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 19h
Merged PRs (30d)
206

Description

The Harbor (`trl.experimental.harbor`) and OpenReward (`trl.experimental.openreward`) integrations predate #6238 (environment-owned reward) and #6349 (environment-owned dataset). Both now carry boilerplate those PRs make unnecessary:

- each fabricates an empty `prompt` column in `spec.train_dataset` (`[{"role": "user", "content": ""}]` per row);
- each ships a `reward_funcs` (`_outcome_reward_func` / `_outcome_only_reward_func`) that only reads `env.reward`.

Both PRs are ergonomics (no new capability), but adopting them simplifies both specs and aligns them with the environment-owned model the trainer now expects.

## Proposed — both specs

**1. Environment-owned reward (#6238).** Replace the `reward` property + `_outcome_reward_func` with a reserved `get_reward()` method on the env (`HarborEnv` / OpenReward `_RolloutEnvironment`). The trainer auto-registers it as a reward source (named after the env class, weight 1) and makes `reward_funcs` optional. Bonus: because `get_reward` is reserved from tool discovery, the probe-time guard in Harbor's `reward` property (`if self._env is None: return 0.0` — needed only because `inspect.getmembers` *evaluated* the property on a fresh env) can be removed; `get_reward()` is never called during discovery.

- *Multireward caveat:* `get_reward()` returns a single `float`; multi-signal setups (e.g. a correctness / submission / tool-efficiency suite) keep explicit `reward_funcs`. This is the default single-outcome path only.

**2. Environment-owned dataset (#6349, once merged).** Drop the fabricated empty `prompt` column — the prompt is now optional and `reset()` supplies it.

- **Keep the dataset task-driven** (one `task_dir` / task per row). *Not* going dataset-less: Harbor/OpenReward need the `G` rollouts of a GRPO group to share the same task for a meaningful group baseline; environment self-sampling would break it (see the group-state note on #6349). Only the empty prompt column goes.

## Composition bonus (#6002)

The per-env-class `get_reward` source is scoped by env type (`e.get_reward() if type(e) is _env_type else None`), so composing Harbor + OpenReward as a `dict` `environment_factory` gives each its own reward column automatically — no manual reward wiring for multi-environment routing.

## Scope

- `trl/experimental/harbor/{_env.py,_spec.py}`
- `trl/experimental/openreward/{environment.py,_spec.py}`
- both integration docs guides

## Non-goals

- Multireward stays on explicit `reward_funcs`.
- No dataset-less mode for these integrations (group baseline).

I own both integrations and will take this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.