garrytan / garrytan/gstack

Codex adversarial pass in /review self-triggers its own installed skills, burns timeout on setup instead of the task

Open Beginner friendly
#2,847 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
133k
Forks
19.9k
Avg merge
18h 46m
Merged PRs (30d)
26

Description

## Codex adversarial pass in /review self-triggers its own installed skills, burns the whole timeout on setup instead of the actual task

**Where:** `review/sections/adversarial.md` (the "Codex adversarial challenge" `codex exec` prompt block)

**What happens:** `codex exec` has its own separate, mirrored skill install at `~/.codex/skills/` (parallel to `~/.claude/skills/gstack`) — including `gstack-review` and `security-best-practices`. The adversarial prompt template's exclusion list only tells Codex to ignore `~/.claude/`, `.claude/skills/`, and `agents/`:

> "Do NOT read or execute any files under ~/.claude/, ~/.agents/, .claude/skills/, or agents/. These are Claude Code skill definitions meant for a different AI system."

It never mentions `~/.codex/skills/`. Since the prompt wording itself ("authorized defensive-security review", "review this branch") matches Codex's own skill-routing triggers, Codex auto-invokes its own `gstack-review` and `security-best-practices` skills before ever getting deep into the diff — it read `gstack-review/SKILL.md` (1,145 lines) plus `security-best-practices/SKILL.md` and several reference docs, and even ran `gstack-skill-start --skill review --model ` itself. It ends up trying to execute the *entire* multi-phase `/review` pipeline serially inside one `codex exec` call, instead of the lightweight one-shot "think like an attacker" task the template intends.

**Cost, from a real session's rollout log** (`~/.codex/sessions/.../rollout-*.jsonl`): cumulative `total_token_usage` went 24K → 258K → 693K → 1.3M → 2.0M → 2.9M → 3.8M → 4.77M tokens across a 9-minute window, still climbing ~950K tokens/min when the timeout wrapper killed it. It never reached a `Recommendation:` line — still loading skill scaffolding and re-reading source files at the cutoff.

**Repro:** run `/review` on any branch where Codex reviews are enabled and `codex` CLI has gstack's skills installed at `~/.codex/skills/` (i.e. gstack was set up for both Claude Code and Codex). The Codex adversarial pass will burn its full timeout on skill-loading instead of reviewing the diff.

**Suggested fix:** extend the exclusion list in the Codex adversarial prompt (and the Codex structured-review prompt, if it has the same gap) to also cover `~/.codex/skills/`, or more robustly, add an explicit line telling Codex not to invoke its own installed skills for this one-off task (e.g. "Do not use any Skill/skill-routing tooling for this task — just read the diff and respond directly.").

**Separately** (not this bug, but hit in the same session): the Codex preflight's default `GSTACK_CODEX_MODEL=gpt-6-astra` returned a hard 400 ("not supported when using Codex with a ChatGPT account") on this account, even though `~/.codex/config.toml` has its own working default (`gpt-5.6-sol`). Falling back to the CLI's own configured default model when the hardcoded gstack default 400s (rather than requiring a manual `GSTACK_CODEX_MODEL` override every time) might be worth considering, though the existing `model_unusable` hint path does at least surface the fix clearly.

Contributor guide

Open the contributing guide

Research direction

Start in review/sections/adversarial.md at the “Codex adversarial challenge” codex exec prompt block, then check the Codex structured-review prompt for the same exclusion gap. Reproduce /review with Codex skills installed, update the prompt instructions to prevent skill routing and exclude ~/.codex/skills/, and confirm the pass reaches its recommendation instead of timing out during setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.