cloudflare / cloudflare/pingora

SSE (Server-Sent Events) response body not flushed to downstream on macOS

Open
#841 3 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Rust
Stars
27.4k
Forks
1.7k
Avg merge
6h 22m
Merged PRs (30d)
3

Description

## Description

When proxying SSE (Server-Sent Events) responses through Pingora on macOS, the response body chunks are received from upstream but never flushed to the downstream client. The same setup works correctly on Linux.

## Environment

- **macOS**: Darwin 24.6.0 (Apple Silicon, aarch64)
- **Linux**: Ubuntu 22.04 x86_64 (works correctly)
- **Pingora version**: 0.8.0
- **Rust**: stable

## Steps to Reproduce

1. Set up a simple upstream server that serves SSE responses (`Content-Type: text/event-stream`, `Transfer-Encoding: chunked`)
2. Configure Pingora as a reverse proxy forwarding to this upstream
3. Send a GET request to the proxy endpoint from macOS

## Expected Behavior

SSE events should be streamed to the client in real-time as they arrive from upstream.

## Actual Behavior (macOS only)

- Pingora successfully connects to upstream and receives the response headers
- Upstream body chunks are received (confirmed via `response_body_filter` logging)
- However, **no data reaches the downstream client** — curl shows no output
- After curl timeout, the connection is reset with: `Fail to proxy: Downstream ConnectionClosed ... Prematurely before response body is complete`
- Debug logs show the downstream socket's BufWriter has `written: 0` despite data being in the buffer

## Debug Log Evidence

```
[DEBUG] Detected streaming response: text/event-stream
[DEBUG] Streaming chunk: 937 bytes, eof: false ← data received from upstream
[DEBUG] Streaming chunk: 318 bytes, eof: false ← more data received
[ERROR] Fail to proxy: Downstream ConnectionClosed
context: Prematurely before response body is complete
status: 0 ← 0 bytes written downstream
```

The BufWriter state at connection close:
```
BufStream { inner: BufReader { reader: BufWriter {
writer: ..., buffer: 0/1460, written: 0 ← 1460 bytes buffered, 0 written
}}}
```

## Analysis

The issue appears to be in the HTTP/1.1 downstream write path. Pingora's `do_write_chunked_body` in `body.rs` does call `stream.flush()` after each chunk write. However, on macOS, the data appears to remain in the BufWriter's buffer (1460 bytes = typical TCP MSS) without being flushed to the socket.

This suggests a platform-specific difference in how `AsyncWrite::flush()` behaves on macOS vs Linux TCP sockets, possibly related to:
- TCP_NODELAY not being set on the downstream socket
- macOS TCP stack buffering behavior differences
- BufWriter not propagating flush to the underlying TCP stream

## Workarounds Attempted (none worked on macOS)

- `session.set_keepalive(None)`
- Custom `response_body_filter` returning `Ok(None)` (no delay)
- Custom `upstream_response_filter` setting chunked encoding headers
- Removing all custom filters (bare Pingora proxy) — still fails on macOS

## Impact

- Normal HTTP requests (GET, POST with complete responses) work fine on macOS
- Only streaming/long-lived responses (SSE, potentially WebSocket) are affected
- Linux works perfectly — this is macOS-only

## Notes

macOS is commonly used for development. While Pingora is primarily designed for Linux production deployments, macOS development support would significantly improve the developer experience.

---
*Reported by siyuan — discovered while building gateway-rs*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.