Hook command runner buffers unbounded stdout/stderr in memory

Start in codex-rs/hooks/src/engine/command_runner.rs around lines 64-111 and reproduce the issue with the provided spam_hook.py and hook configuration while observing RSS. Define the runtime output limits with maintainers, then verify bounded stdout and stderr, malformed UTF-8 handling, concurrent noisy handlers, and cleanup using the listed acceptance tests.

What variant of Codex are you using?

Codex CLI/TUI hooks on current main (49025589b0216b876b1a6a20977536c7d55cdb8b). The same hook engine is shared by other surfaces.

What issue are you seeing?

The command-hook runner buffers the complete stdout and stderr streams in memory until the handler exits or times out. There is no byte or character limit on either stream.

Current code pipes both streams and awaits child.wait_with_output(), then creates lossy UTF-8 String copies:

https://github.com/openai/codex/blob/49025589b0216b876b1a6a20977536c7d55cdb8b/codex-rs/hooks/src/engine/command_runner.rs#L64-L111

A noisy, broken, or plugin-provided hook can therefore make Codex memory usage scale with the amount of output produced before process exit. Concurrent hook handlers amplify the peak. The configured timeout bounds elapsed time, but it does not bound bytes produced during that interval.

Minimal reproduction

Register a command hook whose script writes a configurable amount of data:

# spam_hook.py
import os
import sys

mib = int(os.environ.get("HOOK_OUTPUT_MIB", "64"))
chunk = b"x" * (1024 * 1024)
for _ in range(mib):
    sys.stdout.buffer.write(chunk)

{
  "hooks": {
    "UserPromptSubmit": [{
      "hooks": [{
        "type": "command",
        "command": "python spam_hook.py",
        "timeout": 30
      }]
    }]
  }
}

Trigger the event while observing Codex RSS. Increasing HOOK_OUTPUT_MIB increases retained output proportionally. The payload is invalid hook JSON, but it is not parsed or rejected until after the complete stream has been accumulated.

The same behavior applies to stderr, and String::from_utf8_lossy(...).to_string() can add another allocation after collection.

Expected behavior

Hook process output should have a runtime-owned hard bound independent of handler timeout and additionalContextLimit.

A safe contract would:

• read stdout and stderr concurrently with fixed byte/character ceilings;
• terminate the handler when stdout exceeds the protocol limit;
• drain stderr without retaining it indefinitely, keeping only a small bounded diagnostic tail if needed;
• reject malformed UTF-8 rather than silently normalizing protocol bytes;
• report a bounded, content-free failure reason;
• ensure overflow/timeout cleanup terminates the relevant process tree.

Suggested acceptance tests

• a handler producing output just below the limit succeeds;
• output above the limit is terminated and reported without retaining the full stream;
• unbounded stderr does not cause unbounded RSS growth;
• malformed UTF-8 is rejected deterministically;
• several concurrent noisy handlers remain within an aggregate memory bound;
• overflow and timeout leave no surviving child process holding the pipes open.

Related but distinct: #4337 covers process-tree cleanup for shell tool calls, and #21233 reports general Codex memory growth. I could not find an existing issue for the hook command runner's unbounded stdout/stderr capture.

I can prepare a focused implementation and regression tests if maintainers confirm the desired limits and invite a PR.