d0ugal / d0ugal/graith

Investigate and reduce terminal-owned attach input latency

Open
#1,955 1 comment 0 reactions 0 assignees View on GitHub
size: M
Dominant language
Go
Stars
2
Forks
0
Avg merge
5h 49m
Merged PRs (30d)
189

Description

## Problem

Interactive terminal-owned attach can feel like local input has noticeable delay even when the CLI and daemon are on the same machine.

## Short Hot Path

```text
keypress
-> client raw stdin read/filter
-> client data frame over daemon socket
-> daemon writes to PTY master
-> child process consumes key and writes terminal output
-> daemon PTY read loop updates scrollback + daemon screen model
-> terminal-owned attach sends repaint signal/snapshot
-> client renders snapshot/delta to the local terminal
```

The important distinction from raw attach/tmux-style streaming: terminal-owned attach does not render raw PTY bytes directly. It renders from the daemon-maintained screen model.

## Root Cause Found

The dominant measured delay was not socket framing or JSON encoding. It was lock contention in the PTY session read loop.

Before the fix, `Session.readLoop` held `s.mu.RLock()` while blocked in `unix.Poll(..., 100)`. `ScreenSnapshot` / `ScreenSnapshotDelta` need `s.mu.Lock()` to capture and diff the daemon screen model. That meant a snapshot request triggered by fresh PTY output could wait behind the idle read-loop poll, adding up to about 100 ms before the client could render the echoed character.

## Evidence

Opt-in diagnostic command:

```bash
GRAITH_INPUT_LATENCY_DIAGNOSTIC=1 GRAITH_INPUT_LATENCY_SAMPLES=20 go test -v ./internal/daemon -run TestTerminalOwnedAttachInputLatencyDiagnostic -count=1
```

Baseline before the fix:

```text
idle-45ms visible latency: min=100.53475ms median=101.34775ms p95=110.006166ms max=110.33525ms samples=20
idle-45ms output hint latency: min=138µs median=183.084µs p95=9.47675ms max=12.929167ms hints=20
burst-5ms visible latency: min=100.541334ms median=101.316083ms p95=103.232916ms max=110.212792ms samples=20
burst-5ms output hint latency: min=117.25µs median=172.5µs p95=2.637333ms max=14.155875ms hints=20
```

After releasing the session mutex before the blocking poll:

```text
idle-45ms visible latency: min=226.75µs median=297.583µs p95=10.727583ms max=11.292708ms samples=20
idle-45ms output hint latency: min=116µs median=154.25µs p95=10.494ms max=11.141041ms hints=20
burst-5ms visible latency: min=137.959µs median=160.25µs p95=274.5µs max=474.875µs samples=20
burst-5ms output hint latency: min=65.292µs median=74.416µs p95=135.417µs max=145.458µs hints=20
```

Repaint encoding benchmark, for scale:

```bash
go test ./internal/client -run '^$' -bench 'BenchmarkTerminalOwnedAttachFullRepaintFrameBytes' -benchmem -count=5
```

Observed on Apple M5:

```text
80x24 full repaint encode: ~12.6-13.7 us/op, 2.1 KiB wire payload
120x40 full repaint encode: ~30.0-31.8 us/op, 5.0 KiB wire payload
160x48 full repaint encode: ~45.8-47.3 us/op, 7.8 KiB wire payload
240x72 full repaint encode: ~110-127 us/op, 17.3 KiB wire payload
```

## Why tmux can feel fast over a network

tmux also has a server-side terminal model, but attached clients receive pushed output/screen updates from the server's event loop. It avoids a lock-held idle poll in the repaint path and avoids requiring the client to ask for a fresh screen snapshot after every repaint signal. Over SSH, the network adds latency, but the architecture still has fewer local scheduling/lock stalls in the echo path.

## Current Fix Direction

Near-term fix:

- Capture the PTY fd under `Session.mu`, release the lock while blocking in `poll(2)`, then revalidate before reading.
- Add a regression test proving `ScreenSnapshot` can complete while `readLoop` is parked in poll.
- Keep the opt-in latency diagnostic for local evidence gathering without touching a user's live daemon or sessions.

Future work:

- Consider daemon-pushed terminal-owned deltas so the client does not need the extra hint -> request -> response turn.
- Keep coalescing/backpressure for paste and bursty output.
- Separately measure real terminal write/render time outside the daemon harness.

Expanded call path

```text
client attach setup
internal/client/terminal_owned_attach.go runTerminalOwnedPassthrough
internal/client/passthrough.go RunPassthrough/runPassthroughLoop

client key input
internal/client/passthrough.go stdin.Read
internal/client/passthrough.go processKittyPrefix / chrome mouse translation / dragArrow / terminalOwnedInput router
internal/client/passthrough.go prefix and bracketed-paste scan
internal/client/client.go SendData
internal/protocol/frame.go FrameWriter.WriteFrame(channel=data)

daemon input
internal/daemon/handler.go ChannelData attached-session gate
internal/daemon/handler.go pty.WriteInput(frame.Payload)
internal/pty/session.go Session.WriteInput / writeInputLocked / Ptmx.Write

PTY output and screen model
child process writes PTY output
internal/pty/session.go readLoop
internal/pty/session.go Ptmx.Read
internal/pty/session.go Scrollback.Write
internal/pty/session.go writeScreenLocked
internal/pty/session.go attached writer fanout

terminal-owned repaint
internal/daemon/attach_backpressure.go boundedAttachDataWriter.Write / enqueueCoalesced
internal/client/passthrough.go startDemux / terminal-owned data frame
internal/client/passthrough.go requestSnapshot
internal/client/passthrough.go terminalOwnedSnapshotMinInterval
internal/daemon/handler.go screen_snapshot
internal/daemon/handler_query.go handleScreenSnapshot
internal/pty/render.go ScreenSnapshotDelta / renderSnapshotDelta
internal/client/passthrough.go screen_snapshot_response
internal/client/terminal_owned_attach.go writeTerminalOwnedScreenSnapshotWithChrome
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/pty/session.go at Session.readLoop, Ptmx.Read, and the screen snapshot paths; review how the session mutex surrounds the blocking poll. Run the named internal/daemon latency diagnostic and add a regression test showing ScreenSnapshot can complete while readLoop is polling. Done means the mutex is not held during the blocking poll and the snapshot test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.