Windows sandbox wrapper silently drops helper stdout chunks (>2 MiB): image_gen/view_image fail on reference images larger than ~1.5 MB
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.5k
- PR merge metrics
- PR metrics pending
Description
What version of the Codex App are you using?
Codex app 26.915.4065.0 (bundled codex-cli 0.155.0-alpha.9.2), sandbox = "unelevated". The code is unchanged on main and rust-v0.156.0-alpha.9.
What platform is your computer?
Windows 10 Pro 10.0.19045, x64
What issue are you seeing?
image_gen with referenced_image_paths intermittently fails before any request is sent:
unable to read referenced image at …: failed to encode or decode fs sandbox helper message: expected ident at line 1 column 2… expected value at line 1 column 1unable to process referenced image at …: Format error decoding Png: Unknown filter method 255
The files are valid (stable hashes, they decode fine, and the helper alone returns them intact). Only files whose helper response exceeds 2,097,152 bytes are affected (raw file larger than about 1.57 MB). Restarting the app does not help.
Cause (from reading the source at tag rust-v0.155.0-alpha.9.2)
SandboxedFileSystem::read_file receives the whole file as one base64 JSON line on the helper's stdout. On Windows that stdout passes through codex.exe --run-as-windows-sandbox:
read_handle_loopreads the child pipe in 8 KiB chunks on an OS thread and callsoutput_tx.send(chunk)on abroadcast::channel(256)(unified_exec/backends/legacy.rs;elevated.rs/windows_common.rsdo the same).broadcast::Sender::sendnever waits and evicts the oldest value when the queue is full.- The consumer in
codex_utils_pty::spawn_from_driverhandlesRecvError::Lagged(_) => continue, so evicted chunks vanish silently. - The wrapper runs a
new_current_threadruntime, so the consumer task only runs when the main thread is scheduled. The reader thread drains about 330 chunks in a few milliseconds; a slightly late wake-up loses up tochunks - 256chunks.
Losing the head yields a line that starts mid-base64 (expected ident when it starts with t / f / n, otherwise expected value). Losing a middle span removes a multiple of 8192 bytes, which keeps the base64 valid but corrupts the PNG.
What steps can reproduce the bug?
In the app: call image_gen with two or three referenced_image_paths where at least one PNG is larger than ~1.6 MB, under the Windows sandbox. It fails intermittently (three failures in one day of use here, with many successes using the same files in between). It appears to be a scheduling race, so it is presumably more likely under CPU load; I have not measured the rate.
Without image generation:
Feeding the real helper output (8 KiB reads) through a model of the relay (the receiver stalls after k chunks, then resumes at the oldest retained chunk) reproduces the exact three messages for the three real incidents and files, and never fails for files below 256 chunks. A relay with backpressure is intact for every k.
What is the expected behavior?
The wrapper forwards the helper's stdout losslessly, or fails loudly instead of handing truncated data to the parser.
Proposed fix
Apply backpressure in the reader threads instead of let _ = tx.send(..): wait while tx.len() >= capacity && tx.receiver_count() > 0. A patch with two regression tests is attached (not compiled locally). Alternatives: make Lagged a hard error so corruption is never silent, and/or have fs/readFile use the existing fs/open handle-duplication path for large files.
Additional information
The root-cause analysis and the patch below were produced with an AI coding assistant (Claude Code) working from the public source and local measurements; I hit the bug, the assistant traced it. The patch has not been compiled or tested (no Rust toolchain on this machine), so please treat it as a sketch of the intended change. One behavior change to review: a session whose output nobody consumes would now block the child on its pipe instead of silently discarding output.
Proposed patch against rust-v0.155.0-alpha.9.2
diff --git a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/elevated.rs b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/elevated.rs
index 24a5494..e6a457e 100644
--- a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/elevated.rs
+++ b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/elevated.rs
@@ -1,3 +1,4 @@
+use super::windows_common::OUTPUT_CHANNEL_CAPACITY;
use super::windows_common::finish_driver_spawn;
use super::windows_common::make_runner_resizer;
use super::windows_common::start_runner_pipe_writer;
@@ -229,11 +230,11 @@ pub(crate) async fn spawn_windows_sandbox_session_elevated_for_permission_profil
let (pipe_write, pipe_read) = transport.into_files();
let (writer_tx, writer_rx) = mpsc::channel::<Vec<u8>>(128);
- let (stdout_tx, stdout_rx) = broadcast::channel::<Vec<u8>>(256);
+ let (stdout_tx, stdout_rx) = broadcast::channel::<Vec<u8>>(OUTPUT_CHANNEL_CAPACITY);
let stderr_rx = if tty {
None
} else {
- Some(broadcast::channel::<Vec<u8>>(256))
+ Some(broadcast::channel::<Vec<u8>>(OUTPUT_CHANNEL_CAPACITY))
};
let (exit_tx, exit_rx) = oneshot::channel::<i32>();
diff --git a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/legacy.rs b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/legacy.rs
index ca13e19..8bb3238 100644
--- a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/legacy.rs
+++ b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/legacy.rs
@@ -1,4 +1,6 @@
+use super::windows_common::OUTPUT_CHANNEL_CAPACITY;
use super::windows_common::finish_driver_spawn;
+use super::windows_common::send_output_chunk;
use crate::conpty::ConptyInstance;
use crate::conpty::spawn_conpty_process_as_user;
use crate::desktop::LaunchDesktop;
@@ -165,7 +167,7 @@ fn spawn_output_reader(
output_tx: broadcast::Sender<Vec<u8>>,
) -> std::thread::JoinHandle<()> {
read_handle_loop(output_read, move |chunk| {
- let _ = output_tx.send(chunk.to_vec());
+ send_output_chunk(&output_tx, chunk.to_vec());
})
}
@@ -384,11 +386,11 @@ pub(crate) async fn spawn_windows_sandbox_session_legacy(
)?;
let (writer_tx, writer_rx) = mpsc::channel::<Vec<u8>>(128);
- let (stdout_tx, stdout_rx) = broadcast::channel::<Vec<u8>>(256);
+ let (stdout_tx, stdout_rx) = broadcast::channel::<Vec<u8>>(OUTPUT_CHANNEL_CAPACITY);
let stderr_rx = if tty {
None
} else {
- Some(broadcast::channel::<Vec<u8>>(256))
+ Some(broadcast::channel::<Vec<u8>>(OUTPUT_CHANNEL_CAPACITY))
};
let (exit_tx, exit_rx) = oneshot::channel::<i32>();
diff --git a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common.rs b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common.rs
index 05a8778..477a39d 100644
--- a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common.rs
+++ b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common.rs
@@ -14,6 +14,7 @@ use codex_utils_pty::TerminalSize;
use codex_utils_pty::WindowsTtyInputNormalizer;
use codex_utils_pty::spawn_from_driver;
use std::fs::File;
+use std::time::Duration;
use tokio::sync::broadcast;
use tokio::sync::mpsc;
use tokio::sync::oneshot;
@@ -26,6 +27,26 @@ pub(crate) fn finish_driver_spawn(driver: ProcessDriver, stdin_open: bool) -> Sp
spawned
}
+/// Output chunks a backend may queue before its reader thread waits for the consumer.
+pub(crate) const OUTPUT_CHANNEL_CAPACITY: usize = 256;
+
+/// Sends one output chunk without letting a slow consumer lose data.
+///
+/// `broadcast::Sender::send` never waits. Once the queue is full it evicts the oldest chunk, and the
+/// stream reader in `codex_utils_pty` skips the resulting `Lagged` error. A reader thread can drain a
+/// child pipe much faster than the consumer task is scheduled, so any output larger than the queue
+/// (256 chunks of 8 KiB, about 2 MiB) could silently lose whole chunks. Callers that parse the stream,
+/// such as the fs sandbox helper protocol, then see truncated JSON or corrupted base64 payloads.
+///
+/// Waiting here restores pipe-style backpressure. The wait ends when the last receiver is dropped so
+/// an abandoned session cannot park the reader thread forever.
+pub(crate) fn send_output_chunk(output_tx: &broadcast::Sender<Vec<u8>>, chunk: Vec<u8>) {
+ while output_tx.len() >= OUTPUT_CHANNEL_CAPACITY && output_tx.receiver_count() > 0 {
+ std::thread::sleep(Duration::from_millis(1));
+ }
+ let _ = output_tx.send(chunk);
+}
+
pub(crate) fn start_runner_pipe_writer(
mut pipe_write: File,
) -> std::sync::mpsc::Sender<FramedMessage> {
@@ -110,13 +131,13 @@ pub(crate) fn start_runner_stdout_reader(
if let Ok(data) = decode_bytes(&payload.data_b64) {
match payload.stream {
OutputStream::Stdout => {
- let _ = stdout_tx.send(data);
+ send_output_chunk(&stdout_tx, data);
}
OutputStream::Stderr => {
if let Some(stderr_tx) = stderr_tx.as_ref() {
- let _ = stderr_tx.send(data);
+ send_output_chunk(stderr_tx, data);
} else {
- let _ = stdout_tx.send(data);
+ send_output_chunk(&stdout_tx, data);
}
}
}
diff --git a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common_tests.rs b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common_tests.rs
index 1a957e1..153825e 100644
--- a/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common_tests.rs
+++ b/codex-rs/windows-sandbox-rs/src/unified_exec/backends/windows_common_tests.rs
@@ -32,3 +32,43 @@ fn runner_result_reporting_preserves_exit_and_closed_pipe_results() -> Result<()
}
Ok(())
}
+
+#[test]
+fn send_output_chunk_waits_for_a_slow_receiver_instead_of_dropping_chunks() {
+ let (output_tx, mut output_rx) = broadcast::channel::<Vec<u8>>(OUTPUT_CHANNEL_CAPACITY);
+ let chunk_count = OUTPUT_CHANNEL_CAPACITY * 4;
+ let sender = std::thread::spawn(move || {
+ for index in 0..chunk_count {
+ send_output_chunk(&output_tx, (index as u32).to_be_bytes().to_vec());
+ }
+ });
+
+ // Let the sender reach the full queue before the receiver starts, as a late-scheduled
+ // consumer task would.
+ std::thread::sleep(Duration::from_millis(50));
+ let mut received = Vec::new();
+ loop {
+ match output_rx.blocking_recv() {
+ Ok(chunk) => received.push(chunk),
+ Err(broadcast::error::RecvError::Closed) => break,
+ Err(broadcast::error::RecvError::Lagged(skipped)) => {
+ panic!("receiver lost {skipped} output chunks")
+ }
+ }
+ }
+ sender.join().expect("sender thread");
+
+ let expected = (0..chunk_count)
+ .map(|index| (index as u32).to_be_bytes().to_vec())
+ .collect::<Vec<_>>();
+ assert_eq!(received, expected);
+}
+
+#[test]
+fn send_output_chunk_does_not_wait_after_the_last_receiver_is_dropped() {
+ let (output_tx, output_rx) = broadcast::channel::<Vec<u8>>(OUTPUT_CHANNEL_CAPACITY);
+ drop(output_rx);
+ for _ in 0..OUTPUT_CHANNEL_CAPACITY * 2 {
+ send_output_chunk(&output_tx, vec![0]);
+ }
+}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in codex-rs/windows-sandbox-rs/src/unified_exec/backends/legacy.rs, elevated.rs, and windows_common.rs, then inspect the consumer in codex_utils_pty::spawn_from_driver. Run the existing Windows sandbox tests and the regression cases described in windows_common_tests.rs. Done means helper stdout and stderr remain lossless for outputs over 2 MiB, and abandoned sessions do not wait indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- devtools, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100