ByteDance-Seed / ByteDance-Seed/EdgeBench
**How can reasoning effort be explicitly configured for GPT-5.5 with the Codex harness?**
- Dominant language
- Python
- Stars
- 444
- Forks
- 19
- Avg merge
- 6m
- Merged PRs (30d)
- 1
Description
I am running EdgeBench with the official Codex configuration:
```yaml
model:
model: gpt-5.5
defaults:
agent: codex
```
I would like to explicitly specify the model’s reasoning effort, such as `low`, `medium`, or `high`, to ensure that evaluation results are reproducible and comparable.
### Findings from the source code
After reviewing the current implementation, I found the following:
1. `sforge/harness/agent/codex.py` installs `@openai/codex@0.130.0`.
2. When `OPENAI_BASE_URL` is set, the harness generates `~/.codex/config.toml` containing:
```toml
model_provider = "sforge-proxy"
model_verbosity = "medium"
model = "${CODEX_MODEL:-gpt-5.5}"
```
However, this configuration does not explicitly set `model_reasoning_effort`. `model_verbosity` appears to control response verbosity rather than reasoning effort.
3. The Codex execution command is:
```bash
codex exec --dangerously-bypass-approvals-and-sandbox "$(cat {prompt_file})"
```
No reasoning-effort option is passed through the command line.
4. `SForgeConfig` exposes `agent_model` and `agent_extra_env`, but I could not find a dedicated reasoning-effort field.
5. `sforge/visualizer/parsers/codex_output.py` parses `reasoning effort` from the Codex session header and stores it as `reasoning_effort`. The example in that parser contains:
```text
reasoning effort: high
```
This only demonstrates that the effective effort is observable in Codex output. It does not confirm which reasoning effort was used for the official GPT-5.5 leaderboard runs, because the example uses a different Codex version and model.
### Questions
1. Is there currently a supported way to configure reasoning effort for GPT-5.5 with the Codex harness, through the experiment YAML, an environment variable, or another configuration option?
2. Which reasoning effort was used for the official GPT-5.5 + Codex leaderboard results: `low`, `medium`, `high`, or a model/CLI default?
3. If the harness relies on the Codex CLI default, could changes in the default behavior across Codex CLI versions affect reproducibility and leaderboard comparability?
4. Would it be possible to expose an explicit option such as:
```yaml
model:
model: gpt-5.5
reasoning_effort: high
```
and map it to either:
```toml
model_reasoning_effort = "high"
```
or:
```bash
codex exec -c model_reasoning_effort=high ...
```
Explicitly recording the effective reasoning effort in the experiment metadata would also help reproduce and compare evaluation results.
---
Contributor guide
Research direction
Start by reading sforge/harness/agent/codex.py and the SForgeConfig handling for agent_model and agent_extra_env, then inspect sforge/visualizer/parsers/codex_output.py for how reasoning_effort is recorded. Done means the supported configuration path and leaderboard default are established, with any explicitly selected effective effort captured in experiment metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100