optimizations for websocket streaming

## Use Case

We would like for streaming mode from agave remote server to be as fast as possible. Assume for now that we want to continue using WebSockets as the transport technology.

## Solution

# AGAVE Server-Mode Streaming Optimization Plan

Goal: return rendered images across the WebSocket as fast as possible in server
mode, with low latency in streaming mode. Focus is on **server-side**
optimizations that do not require coordinated client changes (pyclient /
webclient) unless explicitly noted.

## The server-side hot path (per streamed frame)

For each frame in stream mode:

1. `Renderer::render()` runs on the render thread — `agave_app/renderer.cpp`
(~L400–L428):
- `new uint8_t[w*h*4]` heap allocation every frame (4 MB at 1024²)
- `glReadPixels` GL→CPU readback (`GL_BGRA` / `GL_UNSIGNED_INT_8_8_8_8_REV`,
which already matches `QImage::Format_ARGB32` on little-endian — no swizzle)
- `QImage(...).copy()` — full 4 MB copy
- `img.mirrored()` for the OpenGL backend — another 4 MB copy + row flip
2. `emit requestProcessed(...)` with `Qt::BlockingQueuedConnection`
(`agave_app/streamserver.cpp` ~L54–L60) hops to the GUI thread and **blocks
the render thread**.
3. `StreamServer::sendImage()` runs on the **GUI thread**
(`agave_app/streamserver.cpp` ~L250–L302):
- JPEG encode `image.save(..., "JPG", 92)` at 1024² (several ms of CPU) — on
the GUI thread, while the render thread is blocked
- `client->sendBinaryMessage(ba)`

Key structural problem: **JPEG encoding happens on the GUI thread while the
render thread is blocked** (because of `BlockingQueuedConnection`), and all
`THREAD_COUNT` (4) renderer threads serialize their encodes through that single
GUI thread.

---

## Prioritized optimizations (server-side, no client changes needed)

### 1. Move JPEG encoding onto the render thread (biggest win)

Today the render thread emits a `QImage` and stalls while the GUI thread does
the expensive encode *and* the network write. Encode on the render thread
instead:

- Encoding parallelizes across all `THREAD_COUNT` renderers instead of
serializing on the GUI thread.
- `sendImage` then only calls `client->sendBinaryMessage(ba)`, which just queues
into Qt's write buffer and returns fast.
- Keep `BlockingQueuedConnection` for backpressure (prevents a runaway frame
queue), but the blocking window shrinks from *encode + send* to just
*enqueue-send*.
- Bytes on the wire are identical JPEG, so **pyclient and webclient need no
changes**.

Detailed design in the section "Encode-on-render-thread plan" below.

### 2. Eliminate redundant full-frame copies in `render()`

At `agave_app/renderer.cpp` (~L418–L427):

```cpp
std::unique_ptr bytes(new uint8_t[w*h*4]); // 4MB heap alloc every frame
m_fbo->toImage(bytes.get());
QImage img = QImage(...).copy(); // full 4MB copy
if (OpenGL) img = img.mirrored(); // another 4MB copy + flip
```

- Reuse a persistent member buffer instead of `new[]` each frame.
- Build the `QImage` as a **non-owning view** over that buffer (drop `.copy()`).
- The `mirrored()` copy can be folded into the encode step (or avoided by
reading rows flipped). The readback already matches `Format_ARGB32`, so there
is no color swizzle to worry about.

### 3. Adaptive JPEG quality during streaming

Quality is hardcoded to 92 at `agave_app/streamserver.cpp` (~L288). Since stream
mode sends *every accumulation iteration*, early/noisy iterations can be sent at
lower quality (smaller payload, lower latency) and ramped to high quality as the
path tracer converges. Still plain JPEG — no client change. Note: this changes
visual output, so it is a judgment call, not a pure win.

### 4. Verify WebSocket per-message deflate is off

JPEG is already compressed; running permessage-deflate over it burns CPU and
adds latency for ~0 size benefit. Confirm the `QWebSocketServer` isn't
negotiating compression.

---

## Caveats / things already checked

- **TCP_NODELAY (Nagle):** the classic low-latency WebSocket fix, but
`QWebSocket` doesn't expose the underlying `QTcpSocket` / `setSocketOption`, so
it's not directly reachable here. Frames are large (tens–hundreds of KB) so
Nagle matters less than for tiny messages, though the tail segment can still
hit a delayed-ACK stall. Not easily actionable without patching Qt.
- **Client-side (independent, minor):** the pyclient does
`copy.deepcopy(message)` on every received frame at
`agave_pyclient/agave_pyclient/agave_client.py` (~L55–L57) — an extra full
copy of each image. Harmless to fix but unrelated to server latency.

## Optional, larger change (requires client sync)

The only optimization requiring coordinated client changes is switching the wire
codec — e.g. sending raw RGBA for GPU upload, or WebP/AVIF for better
size-at-quality. Bigger change with tradeoffs; the items above give most of the
latency win without touching the clients.

---

# Encode-on-render-thread plan (detailed)

There are **two** consumers of the frame signal, which shapes the plan:

- `StreamServer::sendImage` (server mode) — only needs JPEG bytes for the socket
(`agave_app/streamserver.cpp` ~L250–L302).
- `RenderDialog::onRenderRequestProcessed` / `onFrameDone` (interactive GUI
render) — needs the raw `QImage` for on-screen display (`setImage`) and for
saving to disk in arbitrary formats (`agave_app/renderDialog.cpp` ~L809–L893).

So we cannot swap the signal's payload from `QImage` to `QByteArray` globally —
RenderDialog must keep getting a `QImage`. The plan gates encoding behind a
per-renderer flag and adds a parallel signal.

## How the encoding is actually done

It's the *same Qt JPEG codec call that runs today in `sendImage`*, just
relocated onto the render thread. The current code:

```cpp
QByteArray ba;
QBuffer buffer(&ba);
buffer.open(QIODevice::WriteOnly);
image.save(&buffer, DEFAULT_IMAGE_FORMAT, quality); // qjpeg plugin, thread-safe per call
```

moves into `Renderer::processRequest()`, run once per frame after `render()`
returns the `QImage`. The `qjpeg` plugin encodes each call independently (no
shared state), so running it concurrently on the renderer threads is safe.
Optionally use `QImageWriter` instead of `QImage::save` for more control
(progressive / optimized Huffman), but it's the same codec.

Encoding is pure CPU on the `QImage` — it does **not** need the GL context or the
shared `m_openGLMutex`. `render()` already releases that mutex before returning
(the `MutexContextLocker` is scoped inside `render()`), so while renderer thread
A encodes, threads B/C/D can render. That's the parallelism win versus today's
serialize-on-GUI-thread.

## Where in the loop

In `processRequest()`, encode **after** `m_requestMutex.unlock()` (so new
requests can queue while this frame encodes) and replace/augment the emit at
`agave_app/renderer.cpp` (~L327–L345):

```cpp
m_requestMutex.unlock();

if (!isInterruptionRequested()) {
if (m_encodeFrames && lastReq) {
QByteArray ba;
QBuffer buffer(&ba);
buffer.open(QIODevice::WriteOnly);
img.save(&buffer, m_encodeFormat, m_encodeQuality); // on the render thread
emit requestProcessedEncoded(lastReq, ba); // new signal
} else {
emit requestProcessed(lastReq, img); // unchanged path for RenderDialog
}
// ...existing frameDone / shouldContinue logic...
}
```

## Backpressure stays intact

Keep the `Qt::BlockingQueuedConnection` at `agave_app/streamserver.cpp` (~L60) —
it's what prevents an unbounded frame queue. The render thread's blocking window
shrinks from *encode + validity checks + socket write* down to just *validity
checks + `sendBinaryMessage` (enqueue into Qt's write buffer, returns fast)*.
Encoding already happened before the emit, off the GUI thread.

## Concrete change list

**`renderer.h`**
- Add members: `std::atomic m_encodeFrames{false}`, `QByteArray`/`const
char* m_encodeFormat`, `int m_encodeQuality`.
- Add setter, e.g. `void setFrameEncoding(const char* format, int quality)`.
- Add signal: `void requestProcessedEncoded(RenderRequest* request, QByteArray encoded);`

**`renderer.cpp`**
- In `processRequest()`, branch on `m_encodeFrames` as shown above (both the
streaming `lastReq` path and the non-stream path).

**`streamserver.h` / `streamserver.cpp`**
- Add slot `void sendEncodedImage(RenderRequest* request, QByteArray encoded);`
containing the existing client-validity checks + `client->sendBinaryMessage(encoded)`
+ `delete request` (the tail of today's `sendImage`).
- In `createNewRenderer`: call `r->setFrameEncoding(DEFAULT_IMAGE_FORMAT, 92)`
and connect `requestProcessedEncoded → sendEncodedImage` with
`Qt::BlockingQueuedConnection` instead of `requestProcessed → sendImage`.
- `sendImage` can be removed once nothing uses it (RenderDialog uses its own
slots, not this one).

**RenderDialog** — untouched; still receives `QImage` via `requestProcessed`.

**Clients** — untouched. Wire format is identical JPEG.

## Optional enhancements (not part of the core change)

- **Pipeline encode with next render:** hand the `QImage` to a small per-renderer
encode worker/thread so renderer N+1 can start while frame N encodes. More
throughput, but needs double-buffering and its own backpressure — more
complexity/risk. Do the inline version first and measure.
- **Faster codec:** swap Qt's `qjpeg` for libjpeg-turbo (`turbojpeg`) —
meaningfully faster encode, but adds a dependency.
- **Adaptive quality** during accumulation — separate change, alters output.

---

## Suggested implementation order

1. Encode-on-render-thread (optimization #1) — biggest win, no client changes.
2. Copy elimination in `render()` (optimization #2) — low risk, complements #1.
3. Verify/disable permessage-deflate (optimization #4) — quick check.
4. Adaptive quality (optimization #3) — only if further latency reduction needed;
changes visual output.

---

# Transport alternatives (beyond WebSocket)

Everything above keeps the WebSocket transport. A separate axis for lower
latency is changing *how frames move from server to client*. How much this helps
depends heavily on deployment, and the real win comes less from the wire
protocol itself than from a semantic change it enables: **being allowed to drop
stale frames**.

## Why transport can matter (and when it doesn't)

WebSocket runs over TCP, which guarantees **reliable, in-order** delivery. For
interactive streaming that's the wrong guarantee: if a frame's packet is lost or
the send buffer backs up, TCP head-of-line (HoL) blocking stalls *every* later
frame until the old one is redelivered — even though we'd rather discard the
stale frame and show the newest. On a clean localhost this rarely bites; over a
real network (loss, jitter) it's a major latency source.

So the deployment target drives the decision:

- **Localhost / same machine** (common in current usage): the WebSocket already
rides loopback TCP, which is fast. The encode-on-render-thread + copy
elimination work above helps more than any transport swap. The one bigger win
here is bypassing the network stack entirely (shared memory).
- **Remote / LAN / WAN clients:** this is where a UDP-based, drop-stale-frames
transport gives a real, sometimes dramatic, latency improvement.

## Options, and how they fit the two clients (browser + Python)

| Transport | Latency benefit | Browser client | Python client | Cost |
|---|---|---|---|---|
| **Raw TCP** | Slightly less framing overhead than WS | ❌ not possible in a browser | ✅ | Low, but loses web client — not worth it |
| **Shared memory / local socket** | Huge for co-located (no encode+copy+TCP; could share raw framebuffer) | ❌ sandboxed | ✅ localhost only | Medium; localhost-only |
| **WebTransport (HTTP/3 / QUIC)** | UDP, independent streams + unreliable datagrams → no TCP HoL, drop stale frames | ✅ Chrome/Edge, ⚠️ Firefox | ✅ via `aioquic` | High — Qt has no native server; need a QUIC lib/process |
| **WebRTC data channel** | UDP, unreliable/partial-reliability → drop stale frames, no HoL | ✅ native | ✅ `aiortc` | High — needs signaling + WebRTC stack |
| **WebRTC video track** | All of the above **plus inter-frame (delta) compression + hardware encode/decode** | ✅ native | ✅ `aiortc` | Highest — but the true low-latency answer |

## The standout option: WebRTC video track

This is the architecture behind cloud gaming and Unreal's Pixel Streaming, and
the genuinely lowest-latency path for remote interactive rendering:

- **Inter-frame compression** — instead of an independent JPEG every
accumulation iteration, send a real video stream (H.264 / VP9 / AV1) where
converging frames are mostly deltas. For a path tracer refining the *same*
image over many iterations, consecutive frames are nearly identical, so this
collapses bandwidth dramatically.
- **Hardware encode/decode** — NVENC / VideoToolbox on the server, GPU decode in
the browser, both far faster than CPU JPEG.
- **Built-in jitter buffer, congestion control, and frame dropping** — the
freshest frame wins automatically.

Cost: add a WebRTC stack on the server (e.g. GStreamer `webrtcbin`, libwebrtc,
or a Pion/aiortc sidecar), a signaling channel (reuse the existing WebSocket just
for SDP/ICE handshake), and substantial client changes — the browser renders a
`` element instead of decoding JPEG blobs, and the Python client uses
`aiortc`. `renderlib` stays clean; this lives in the `agave_app` server layer.

## Recommendation by scenario

1. **Mostly remote clients:** the lower-latency plan is **WebRTC with a
hardware-encoded video track**, using the existing WebSocket only for
signaling. Biggest win, biggest effort — and it *does* require coordinated
client changes.
2. **Mostly local clients:** skip transport changes — do encode-on-render-thread
+ copy elimination first, then consider a **shared-memory ring buffer** for
the co-located Python client. The browser can't benefit from shared memory,
so its ceiling stays at the WebSocket+JPEG path.
3. **Middle ground, least disruption:** keep WebSocket for signaling/commands but
move *frames* to a **WebRTC data channel** (or WebTransport datagrams)
carrying the same JPEG bytes. Loses inter-frame compression but gains
drop-stale-frame + no TCP HoL, with a smaller rewrite than a full media
pipeline.