Test network error branches with shimforge
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
Disclosure: I maintain shimforge (MIT), the crate this issue suggests.
Problem
Several places decide whether to retry, or which error to report, from reqwest::Error flags such as is_connect(), is_timeout() and is_body(). reqwest::Error has no public constructor, so a test can only get one of these errors from a real failure: a server that stalls, or a port that refuses the connection. Most of these branches are untested today:
| Code | What it decides | Tested today |
|---|---|---|
exec-server/src/client_recovery.rs:871 is_retryable_registry_error |
retry after a registry request fails | status codes and RouteAwareRequestError::Timeout only; not is_connect, is_body or is_decode |
agent-identity/src/lib.rs:213 is_retryable_registration_cause |
retry agent registration | not the HttpError status or transport branches |
codex-api/src/files.rs:196 blob upload error_kind |
timeout / connect / body / request / other in the error and log event |
no tests; the timeout is 60 s |
cli/src/doctor/network.rs:149 request_error |
the message codex doctor prints |
none of its 10 messages |
feedback/src/upload.rs:84 |
retry a feedback upload after a network error | no unit tests |
A real server is a poor way to reach these branches. Timeouts cost real time, and connect errors are platform-dependent: on Windows a refused localhost connection takes about 2 s, so a connect test with a short timeout fails with a timeout instead.
Proposal
Add shimforge as a dev-dependency. It redirects calls to a function at run time, for the current test thread only. A test can then start from a real reqwest::Error built without any network access (a URL that does not parse) and mock only the flag it checks. Production code does not change.
Example: connect failures are retried
[workspace.dependencies]
shimforge = "0.1.3"
[dev-dependencies]
reqwest = { workspace = true }
shimforge = { workspace = true }
exec-server/src/client_recovery_tests.rs:
use shimforge::Session;
use shimforge::mock;
/// A real `reqwest::Error` from a URL that does not parse. No request is sent.
fn registry_request_error() -> ExecServerError {
let error = reqwest::Client::new()
.get("not a url")
.build()
.expect_err("the URL does not parse");
ExecServerError::EnvironmentRegistryRequest(
codex_http_client::RouteAwareRequestError::Request(error),
)
}
#[test]
fn recovery_retries_registry_connect_failures() {
let mut session = Session::new();
let is_connect = mock!(
session,
reqwest::Error::is_connect,
fn(&reqwest::Error) -> bool
);
is_connect.expect().once().returns(true);
assert!(is_retryable_registry_error(®istry_request_error()));
}
#[test]
fn recovery_does_not_retry_other_request_errors() {
assert!(!is_retryable_registry_error(®istry_request_error()));
}
Tests for is_body and is_decode have the same shape, and so do the other rows in the table. No server, no waiting, and the same result on every OS.
Costs and limits
- Dev-dependency only; nothing ships in release builds. Runs on Linux, macOS and Windows, on x86-64 and ARM64.
- It patches machine code at run time, so the mocked function must not be inlined into its caller. Codex's
devandci-testprofiles build atopt-level = 0so it's not a problem.
Evidence
I made the same kind of change in another Rust project: mezmo/aura#711. It replaced a wiremock server and a closed-port connect in error-classification tests with this pattern. The tests pass on Linux, macOS and Windows (x86-64 and ARM64) and in that project's Jenkins coverage run. The closed-port test it replaced was already failing on Windows for the 2 s reason above.
I can post the full test code for any row in the table here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the workspace Cargo.toml and the listed test and implementation files: exec-server/src/client_recovery_tests.rs, agent-identity/src/lib.rs, codex-api/src/files.rs, cli/src/doctor/network.rs, and feedback/src/upload.rs. Review each error branch and existing tests, then run the relevant test suites; done means the listed reqwest error paths are covered deterministically without network waits or platform-specific behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100