RFC: End a hands-free turn by asking "did they finish?", not by timing silence

## The ask

After today's 300 ms silence trigger, run a small **local** model on the
audio. Wake the agent only if the model says the speaker finished. If it
says they are still going, keep listening.

Do not change the silence number. Do not touch push-to-talk. No cloud.
No new wire format.

**Status:** proposal. Flagged prototype and a measurement on a laptop
before anything becomes default. No code.

## Today vs proposed

Left is what Buzz does now. Right is this RFC. Grey is unchanged.
Green is the only new step. Red is the premature wake we stop.

```mermaid
flowchart LR
subgraph T["Today"]
direction TB
t1[You pause mid-thought] --> t2[300 ms silence]
t2 --> t3[Flush and wake the agent]
end
subgraph P["Proposed"]
direction TB
p1[You pause mid-thought] --> p2[300 ms silence]
p2 --> p3{Local model}
p3 -->|HOLD still going| p4[Keep listening]
p3 -->|SHIFT finished| p5[Flush and wake]
end
classDef bad fill:#fdecea,stroke:#c0392b,color:#000
classDef good fill:#e8f8f5,stroke:#1e8449,color:#000
classDef same fill:#f4f6f7,stroke:#7f8c8d,color:#000
class t3 bad
class p3,p4,p5 good
class t1,t2,p1,p2 same
```

## Why a stopwatch cannot be the turn boundary

A pause is not a finished sentence. The two cases can have **identical
words** and still mean opposite things.

LiveKit's turn-detection writeup uses the pizza example: the transcript
at the pause is the same for "I would like to order one large pizza…"
(about to continue) and "I would like to order one large pizza… and a
garlic bread" (already continued). No amount of reading the words
distinguishes them — the distinction is in how they are delivered
(intonation, pitch, rhythm).
Source: [LiveKit, "Solving end-of-turn detection"](https://livekit.com/blog/solving-end-of-turn-detection).

The academic name for that fork is **SHIFT vs HOLD**. Ekstedt & Skantze
(Voice Activity Projection, Interspeech 2022) state it directly: the
system "should be able to tell whether a mutual silence should be
identified as a SHIFT or a HOLD." A thinking pause is HOLD. A real turn
boundary is SHIFT. Those two are not distinguishable from silence
duration alone.
Paper: [arXiv:2205.09812](https://arxiv.org/abs/2205.09812)
([HTML](https://ar5iv.labs.arxiv.org/html/2205.09812)).

That is the whole argument. Tuning the timer is asking a stopwatch a
question it cannot answer.

## We already tried tuning, in both directions

Buzz's STT worker at `93114c9c6` records two failed attempts
([stt.rs#L160-L171](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L160-L171)):

```rust
/// How many 16 kHz samples of silence before we flush to STT.
/// 300 ms × 16 000 Hz / 256 samples-per-frame ≈ 19 frames.
/// Previous value (28 frames / 450 ms) felt sluggish in conversation.
///
/// This window is a turn-taking quality knob, not a latency lever: an earlier
/// env override (`BUZZ_STT_FLUSH_MS`) let it be lowered to 150 ms, which split
/// natural mid-sentence pauses into separate messages and confused the
/// listening agents. Reverted — the window is fixed at the production value.
const SILENCE_FLUSH_FRAMES: usize = 19;
const VAD_FRAME_SAMPLES: usize = 256;
```

| Tried | Result |
| --- | --- |
| **450 ms** (28 frames) | Felt sluggish in conversation |
| **150 ms** (`BUZZ_STT_FLUSH_MS`) | Split mid-sentence pauses into separate messages; confused listening agents; reverted |
| **300 ms** (19 × 256 samples at 16 kHz ≈ 304 ms) | Value at `93114c9c6` |

The number is not the bug. The signal is the bug. This proposal does
**not** retune `SILENCE_FLUSH_FRAMES`.

## What was surveyed

This is not a two-product glance. The question we asked: *do production
voice agents still use silence duration as the turn boundary, or do they
add a second signal — and is any of that shippable in a local Rust
desktop with no cloud STT?*

**In scope:** named, currently shipping turn-end mechanisms with a public
doc or repo. **Out of this table:** unpublished lab models (except
Ekstedt & Skantze above, which names SHIFT/HOLD; it is not a product).

| System | Signal beyond silence | Local / open? | Verdict for Buzz | Source |
| --- | --- | --- | --- | --- |
| LiveKit Agents TurnDetector v1 | Audio-native EOU probability (semantic + prosody) | `v1-mini` open-weight; full v1 is LiveKit Cloud | Architecture match; SDK is Python/Node, weights under LiveKit Model License — not the copy target | [LiveKit blog](https://livekit.com/blog/solving-end-of-turn-detection) |
| LiveKit TurnDetector 2024 (superseded) | Text transformer on the last 4 turns of transcript | Open-weight on Hugging Face | Text-only; LiveKit replaced it because pauses with identical words are indistinguishable | [LiveKit 2024 blog](https://blog.livekit.io/using-a-transformer-to-improve-end-of-turn-detection) |
| **Pipecat Smart Turn v3** | Whisper-Tiny audio classifier after VAD silence; 8 s window | BSD-2-Clause; ONNX; weights + training + data public | **Copy target** — local, VAD-gated, same inference class as Parakeet | [smart-turn repo](https://github.com/pipecat-ai/smart-turn) |
| Deepgram Flux | EOU baked into STT; `EagerEndOfTurn` / `TurnResumed` | Hosted API in cited docs; no open/local model documented here | Pattern useful; no copyable model in the cited material | [Flux quickstart](https://developers.deepgram.com/docs/flux/quickstart) |
| Deepgram legacy endpointing | Silence / word-timing (`UtteranceEnd`) | Cloud silence heuristic | Same class of bug we already have | [End of speech](https://developers.deepgram.com/docs/understanding-end-of-speech-detection) |
| AssemblyAI Universal-3.5 Pro Streaming | Transcript confidence + min/max silence | Commercial self-hosted container; proprietary; GPU-sized | Not an open desktop model | [Turn detection](https://www.assemblyai.com/docs/streaming/universal-streaming/turn-detection), [self-hosted streaming](https://www.assemblyai.com/docs/streaming/self-hosted-streaming) |
| Cartesia STT Turns | Continuous turn-likelihood vs start / eager_end / end | Cloud, proprietary | Event names (`turn.resume`) are a state-machine blueprint only | [Turns API](https://docs.cartesia.ai/use-the-api/stt/turns) |
| OpenAI Realtime `semantic_vad` | Word-based “done speaking” score scales the wait | Cloud, proprietary | Cloud | [Realtime VAD](https://developers.openai.com/api/docs/guides/realtime-vad) |
| OpenAI Realtime `server_vad` | Silence duration (default 500 ms) | Cloud silence heuristic | Same stopwatch | [Realtime VAD](https://developers.openai.com/api/docs/guides/realtime-vad) |
| Speechmatics real-time | Server-side word-timing silence countdown; optional application-owned semantic layer | Client may supply its own detector; tutorial uses open SmolLM2 | Shipped EOU is still a silence heuristic; DIY semantic pattern only | [Blog](https://blog.speechmatics.com/semantic-turn-detection), [docs](https://docs.speechmatics.com/speech-to-text/realtime/turn-detection) |
| Gemini Live | Automatic server VAD; configurable `silenceDurationMs`; interruption cancellation; `interrupted` event | Cloud, proprietary | Cloud silence/activity detector; not a semantic endpoint model | [Live API capabilities](https://ai.google.dev/gemini-api/docs/live-api/capabilities); [reported barge-in bug](https://discuss.ai.google.dev/t/gemini-3-1-flash-live-preview-live-api-server-never-emits-interrupted-barge-in-when-the-user-is-already-speaking-as-the-models-turn-begins/174346) |

**What the table is for:** show the land was walked. Most surveyed systems
add a model-based signal beyond silence; most implementations are hosted
or proprietary. Pipecat Smart Turn v3 is the only open BSD-2 ONNX option
here designed for local, VAD-gated inference.

## What we would copy

Pipecat Smart Turn v3. Evidence is the project's README at
[`4786657`](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md):

| Claim | Evidence |
| --- | --- |
| Audio classifier on raw PCM after VAD, not a transcript | [README#L7-L26](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L7-L26) |
| Whisper-Tiny backbone, ~8M params | [README#L116-L119](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L116-L119) |
| Local ONNX analyzer `LocalSmartTurnAnalyzerV3` | [README#L70-L80](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L70-L80) |
| Input: 16 kHz mono PCM, **up to 8 seconds**; truncate older audio | [README#L88-L94](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L88-L94) |
| On resume, re-run on the **whole retained recording** (still inside that 8 s window), not just the new segment | [README#L96-L98](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L96-L98) |
| BSD-2-Clause; weights, training code, datasets open | [README#L7-L26](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L7-L26); [Hugging Face: pipecat-ai/smart-turn-v3](https://huggingface.co/pipecat-ai/smart-turn-v3) |

Why this one rather than LiveKit's open `v1-mini`:
Buzz already runs speech-to-text as Parakeet through sherpa-onnx /
ONNX Runtime
([stt.rs#L185-L194](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L185-L194),
[stt.rs#L240-L250](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L240-L250)).
Smart Turn v3 is the same *model format and inference class*. It is
**not** a drop-in Rust crate. Pipecat's local wrapper is Python
([README#L70-L80](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L70-L80);
[`LocalSmartTurnAnalyzerV3` API](https://reference-server.pipecat.ai/en/stable/api/pipecat.audio.turn.smart_turn.local_smart_turn_v3.html)).
The prototype must verify a Rust/ONNX integration path; none was found as
a ready standalone reference.

**Cost (Pipecat's hardware, not ours):** ~10 ms on some CPUs and under
100 ms on most cloud instances
([README#L18](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L18));
~65 ms on a Pipecat Cloud standard 1x instance with
`LocalSmartTurnAnalyzerV3`
([README#L80](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L80)).
Do not quote a Buzz laptop number until it is measured.

## Sequence we would add (nothing else)

1. Earshot VAD already waits for 19 silent frames (~300 ms). **Unchanged.**
2. On that silence, run Smart Turn v3 on the buffered PCM, truncated to
the last ~8 s if longer
([README#L88-L94](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L88-L94)).
**New.**
3. HOLD → keep accumulating; do not wake agents.
4. SHIFT → flush to STT and wake agents, as today.
5. If speech resumes before SHIFT, re-run on the whole retained recording
(again within 8 s), not just the new segment
([README#L96-L98](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L96-L98)).

## Out of scope — and why

| Left out | Why |
| --- | --- |
| Retuning `SILENCE_FLUSH_FRAMES` | Already tried 450 ms (sluggish) and 150 ms (split sentences). The comment at [stt.rs#L160-L171](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L160-L171) is the experiment log. A stopwatch cannot tell SHIFT from HOLD. |
| Push-to-talk mention behaviour | Hands-free is the bug. At `93114c9c6`, a held PTT shortcut never lets silence flush ([stt.rs#L544-L552](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L544-L552)); the falling edge of transmit flushes the buffer ([stt.rs#L310-L325](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L310-L325)). Mixing the two problems hides whether the model helped. |
| New Nostr / ACP wire types | The model sits after local VAD and before the existing STT flush. Nothing on the wire changes. |
| [#3176](https://github.com/block/buzz/issues/3176) batch policy | Queue batching for typed service workflows. Opposite knob from voice turn-taking. |
| Shipping this on by default | Laptop latency and the Rust/ONNX path are unmeasured. This RFC asks for a flagged prototype, not a default. |
| Closing [#3071](https://github.com/block/buzz/issues/3071) / [#3282](https://github.com/block/buzz/issues/3282) | Those are steer-path races (a control signal aimed at no prompt). This RFC is when to flush audio. Different layer. |

## What we do not know yet

| Unknown | Why it matters |
| --- | --- |
| Latency on a typical laptop | Pipecat's 10–65 ms is not a Buzz measurement |
| Non-English speech | README lists 23 languages ([README#L15-L16](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L15-L16)); we have not tested huddle languages |
| 30 s Buzz buffer vs Smart Turn's 8 s window | Buzz keeps up to 30 s of PCM ([stt.rs#L40-L42](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L40-L42), [stt.rs#L458-L464](https://github.com/block/buzz/blob/93114c9c65138397de39729fde0a816eb9f314ab/desktop/src-tauri/src/huddle/stt.rs#L458-L464)). Smart Turn truncates to the last ~8 s ([README#L88-L94](https://github.com/pipecat-ai/smart-turn/blob/4786657e242dfe77dd138699ac564ee074a2a543/README.md#L88-L94)). A long huddle turn can still sit in our buffer while the model only sees the tail. Unmeasured whether that truncation hides HOLD vs SHIFT. |
| Rust integration | Python ONNX wrapper exists; a Buzz-shaped Rust path does not, yet |

## Decision requested

Approve the direction: a **flagged prototype** plus a laptop
measurement, before any change to default hands-free behaviour.
Say go / wait / change the ask.