openai / openai/codex

Test network error branches with shimforge

Open
#46,596 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CLI connectivity enhancement
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

Disclosure: I maintain shimforge (MIT), the crate this issue suggests.

Problem

Several places decide whether to retry, or which error to report, from reqwest::Error flags such as is_connect(), is_timeout() and is_body(). reqwest::Error has no public constructor, so a test can only get one of these errors from a real failure: a server that stalls, or a port that refuses the connection. Most of these branches are untested today:

Code What it decides Tested today
exec-server/src/client_recovery.rs:871 is_retryable_registry_error retry after a registry request fails status codes and RouteAwareRequestError::Timeout only; not is_connect, is_body or is_decode
agent-identity/src/lib.rs:213 is_retryable_registration_cause retry agent registration not the HttpError status or transport branches
codex-api/src/files.rs:196 blob upload error_kind timeout / connect / body / request / other in the error and log event no tests; the timeout is 60 s
cli/src/doctor/network.rs:149 request_error the message codex doctor prints none of its 10 messages
feedback/src/upload.rs:84 retry a feedback upload after a network error no unit tests

A real server is a poor way to reach these branches. Timeouts cost real time, and connect errors are platform-dependent: on Windows a refused localhost connection takes about 2 s, so a connect test with a short timeout fails with a timeout instead.

Proposal

Add shimforge as a dev-dependency. It redirects calls to a function at run time, for the current test thread only. A test can then start from a real reqwest::Error built without any network access (a URL that does not parse) and mock only the flag it checks. Production code does not change.

Example: connect failures are retried

Cargo.toml:

[workspace.dependencies]
shimforge = "0.1.3"

exec-server/Cargo.toml:

[dev-dependencies]
reqwest = { workspace = true }
shimforge = { workspace = true }

exec-server/src/client_recovery_tests.rs:

use shimforge::Session;
use shimforge::mock;

/// A real `reqwest::Error` from a URL that does not parse. No request is sent.
fn registry_request_error() -> ExecServerError {
    let error = reqwest::Client::new()
        .get("not a url")
        .build()
        .expect_err("the URL does not parse");
    ExecServerError::EnvironmentRegistryRequest(
        codex_http_client::RouteAwareRequestError::Request(error),
    )
}

#[test]
fn recovery_retries_registry_connect_failures() {
    let mut session = Session::new();
    let is_connect = mock!(
        session,
        reqwest::Error::is_connect,
        fn(&reqwest::Error) -> bool
    );
    is_connect.expect().once().returns(true);

    assert!(is_retryable_registry_error(&registry_request_error()));
}

#[test]
fn recovery_does_not_retry_other_request_errors() {
    assert!(!is_retryable_registry_error(&registry_request_error()));
}

Tests for is_body and is_decode have the same shape, and so do the other rows in the table. No server, no waiting, and the same result on every OS.

Costs and limits

  • Dev-dependency only; nothing ships in release builds. Runs on Linux, macOS and Windows, on x86-64 and ARM64.
  • It patches machine code at run time, so the mocked function must not be inlined into its caller. Codex's dev and ci-test profiles build at opt-level = 0 so it's not a problem.

Evidence

I made the same kind of change in another Rust project: mezmo/aura#711. It replaced a wiremock server and a closed-port connect in error-classification tests with this pattern. The tests pass on Linux, macOS and Windows (x86-64 and ARM64) and in that project's Jenkins coverage run. The closed-port test it replaced was already failing on Windows for the 2 s reason above.

I can post the full test code for any row in the table here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workspace Cargo.toml and the listed test and implementation files: exec-server/src/client_recovery_tests.rs, agent-identity/src/lib.rs, codex-api/src/files.rs, cli/src/doctor/network.rs, and feedback/src/upload.rs. Review each error branch and existing tests, then run the relevant test suites; done means the listed reqwest error paths are covered deterministically without network waits or platform-specific behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.