openai / openai/codex

History Notes forwards `detail: original` when the selected model does not support it

Open
#41,856 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug CLI tool-calls
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

Background

History Notes can return image attachments from the Codex history backend. These attachments are converted into input_image function-call output items and included in the following Responses request.

The output path preserves detail: "original" even when the selected model has supports_image_detail_original = false.

Affected version
  • Latest main revision tested: 1c1e17782aeb51a5a253997067fa887a9d593cc9
  • Codex CLI: 0.151.0-alpha.7.2
  • OS: Windows 11 Pro for Workstations 10.0.22631, x64
Minimal reproduction

Adapt the existing integration test in codex-rs/ext/history-notes/tests/image_model_input.rs:

  1. Configure the selected model as not supporting original-detail images:

    .with_model_info_override("gpt-5.4", |model_info| {
        model_info.supports_image_detail_original = false;
    })
    
  2. Keep the mocked History Notes backend response containing:

    {
      "data": "<valid 1x1 PNG base64>",
      "mime_type": "image/png",
      "detail": "original"
    }
    
  3. Expect the next Responses request to contain detail: "high", matching the existing sanitize_original_image_detail behavior for models without original-detail support.

  4. Run:

    just test -p codex-history-notes-extension history_images_reach_the_next_model_request
    

The assertion fails consistently because the actual outbound item retains detail: "original":

- "detail": "original"
+ "detail": "high"

The regression assertion failed on both nextest attempts. After restoring the test, the existing crate suite passed: 9 tests run, 9 passed.

Expected behavior

History Notes tool images should follow the selected model's image-detail capability. When supports_image_detail_original is false, detail: "original" should be normalized to the default supported detail (high) before the next Responses request is serialized.

Actual behavior

The next Responses request contains the History Notes image with detail: "original" despite the selected model declaring that original detail is unsupported.

This produces an outbound request that violates the selected model's advertised capability and can make History Notes turns fail on models or providers that reject original-detail image input.

Root cause

HistoryNotesToolOutput::new in codex-rs/ext/history-notes/src/tools.rs copies the backend's image detail directly into FunctionCallOutputContentItem::InputImage:

content.push(FunctionCallOutputContentItem::InputImage {
    image_url: format!("data:{mime_type};base64,{data}"),
    detail,
});

HistoryNotesToolOutput has no model capability context, and its to_response_item implementation returns the payload unchanged.

The general history normalization path strips images for models without image input, but it does not normalize unsupported original detail for image-capable models.

Equivalent model-aware normalization already exists:

  • codex_tools::sanitize_original_image_detail
  • MCP output handling passes original_image_detail_supported and sanitizes before producing the response payload.
  • Code Mode sanitizes runtime image detail using can_request_original_image_detail(turn.model_info()).

The History Notes path bypasses these existing safeguards.

Impact

The issue affects token-budget History Notes sessions when all of the following are true:

  1. The history backend returns an image with detail: "original".
  2. The selected model supports image input but does not support original-detail images.
  3. The model invokes a History Notes tool that returns that attachment.

The affected request is generated deterministically. The scope is limited to History Notes and models without original-detail support.

Potential fix direction

Apply sanitize_original_image_detail to extension tool outputs before they enter history, using the selected turn's ModelInfo. Centralizing this at the extension-tool dispatch boundary would prevent similar capability mismatches in other extension outputs.

Regression coverage should verify:

  • original is downgraded for an image-capable model with supports_image_detail_original = false.
  • original is preserved for a model that supports it.
  • Existing text-only image stripping remains unchanged.
Duplicate review

I searched open and closed Issues, open/draft/closed/merged PRs, recent commits, Discussions, and release-note text using:

  • HistoryNotesToolOutput
  • history notes image detail
  • history backend image
  • supports_image_detail_original history
  • sanitize_original_image_detail history
  • unsupported original image detail

Related but different records:

  • #40595 concerns clipboard images being discarded for text-only models. History Notes images are already stripped by the general history normalizer for text-only models.
  • #24797 concerns remote compaction rejecting the auto variant in persisted history.
  • #19806 concerns token/byte estimation for original-detail images.
  • #41292 introduced History Notes image forwarding but does not test a model with original-detail support disabled.

No existing Issue, PR, commit, or Discussion was found for this History Notes capability mismatch.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with codex-rs/ext/history-notes/src/tools.rs and the existing integration test in codex-rs/ext/history-notes/tests/image_model_input.rs. Read codex_tools::sanitize_original_image_detail and the extension-tool dispatch path to understand where model capability information is available. Run just test -p codex-history-notes-extension history_images_reach_the_next_model_request; done means unsupported original detail is downgraded, supported detail is preserved, and text-only image stripping still passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.