[harbor/openreward] Adopt environment-owned reward (#6238) and dataset (#6349)
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 206
Description
The Harbor (`trl.experimental.harbor`) and OpenReward (`trl.experimental.openreward`) integrations predate #6238 (environment-owned reward) and #6349 (environment-owned dataset). Both now carry boilerplate those PRs make unnecessary:
- each fabricates an empty `prompt` column in `spec.train_dataset` (`[{"role": "user", "content": ""}]` per row);
- each ships a `reward_funcs` (`_outcome_reward_func` / `_outcome_only_reward_func`) that only reads `env.reward`.
Both PRs are ergonomics (no new capability), but adopting them simplifies both specs and aligns them with the environment-owned model the trainer now expects.
## Proposed — both specs
**1. Environment-owned reward (#6238).** Replace the `reward` property + `_outcome_reward_func` with a reserved `get_reward()` method on the env (`HarborEnv` / OpenReward `_RolloutEnvironment`). The trainer auto-registers it as a reward source (named after the env class, weight 1) and makes `reward_funcs` optional. Bonus: because `get_reward` is reserved from tool discovery, the probe-time guard in Harbor's `reward` property (`if self._env is None: return 0.0` — needed only because `inspect.getmembers` *evaluated* the property on a fresh env) can be removed; `get_reward()` is never called during discovery.
- *Multireward caveat:* `get_reward()` returns a single `float`; multi-signal setups (e.g. a correctness / submission / tool-efficiency suite) keep explicit `reward_funcs`. This is the default single-outcome path only.
**2. Environment-owned dataset (#6349, once merged).** Drop the fabricated empty `prompt` column — the prompt is now optional and `reset()` supplies it.
- **Keep the dataset task-driven** (one `task_dir` / task per row). *Not* going dataset-less: Harbor/OpenReward need the `G` rollouts of a GRPO group to share the same task for a meaningful group baseline; environment self-sampling would break it (see the group-state note on #6349). Only the empty prompt column goes.
## Composition bonus (#6002)
The per-env-class `get_reward` source is scoped by env type (`e.get_reward() if type(e) is _env_type else None`), so composing Harbor + OpenReward as a `dict` `environment_factory` gives each its own reward column automatically — no manual reward wiring for multi-environment routing.
## Scope
- `trl/experimental/harbor/{_env.py,_spec.py}`
- `trl/experimental/openreward/{environment.py,_spec.py}`
- both integration docs guides
## Non-goals
- Multireward stays on explicit `reward_funcs`.
- No dataset-less mode for these integrations (group baseline).
I own both integrations and will take this.
Contributor guide
Assessment
This issue has not been assessed yet.