ByteDance-Seed / ByteDance-Seed/EdgeBench

**How can reasoning effort be explicitly configured for GPT-5.5 with the Codex harness?**

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
444
Forks
19
Avg merge
6m
Merged PRs (30d)
1

Description

I am running EdgeBench with the official Codex configuration:

```yaml
model:
model: gpt-5.5

defaults:
agent: codex
```

I would like to explicitly specify the model’s reasoning effort, such as `low`, `medium`, or `high`, to ensure that evaluation results are reproducible and comparable.

### Findings from the source code

After reviewing the current implementation, I found the following:

1. `sforge/harness/agent/codex.py` installs `@openai/codex@0.130.0`.

2. When `OPENAI_BASE_URL` is set, the harness generates `~/.codex/config.toml` containing:

```toml
model_provider = "sforge-proxy"
model_verbosity = "medium"
model = "${CODEX_MODEL:-gpt-5.5}"
```

However, this configuration does not explicitly set `model_reasoning_effort`. `model_verbosity` appears to control response verbosity rather than reasoning effort.

3. The Codex execution command is:

```bash
codex exec --dangerously-bypass-approvals-and-sandbox "$(cat {prompt_file})"
```

No reasoning-effort option is passed through the command line.

4. `SForgeConfig` exposes `agent_model` and `agent_extra_env`, but I could not find a dedicated reasoning-effort field.

5. `sforge/visualizer/parsers/codex_output.py` parses `reasoning effort` from the Codex session header and stores it as `reasoning_effort`. The example in that parser contains:

```text
reasoning effort: high
```

This only demonstrates that the effective effort is observable in Codex output. It does not confirm which reasoning effort was used for the official GPT-5.5 leaderboard runs, because the example uses a different Codex version and model.

### Questions

1. Is there currently a supported way to configure reasoning effort for GPT-5.5 with the Codex harness, through the experiment YAML, an environment variable, or another configuration option?

2. Which reasoning effort was used for the official GPT-5.5 + Codex leaderboard results: `low`, `medium`, `high`, or a model/CLI default?

3. If the harness relies on the Codex CLI default, could changes in the default behavior across Codex CLI versions affect reproducibility and leaderboard comparability?

4. Would it be possible to expose an explicit option such as:

```yaml
model:
model: gpt-5.5
reasoning_effort: high
```

and map it to either:

```toml
model_reasoning_effort = "high"
```

or:

```bash
codex exec -c model_reasoning_effort=high ...
```

Explicitly recording the effective reasoning effort in the experiment metadata would also help reproduce and compare evaluation results.

---

Contributor guide

Open the contributing guide

Research direction

Start by reading sforge/harness/agent/codex.py and the SForgeConfig handling for agent_model and agent_extra_env, then inspect sforge/visualizer/parsers/codex_output.py for how reasoning_effort is recorded. Done means the supported configuration path and leaderboard default are established, with any explicitly selected effective effort captured in experiment metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.