block / block/buzz

Server-side huddle recording with per-speaker audio, so transcripts become channel events agents can read

Open
#2,513 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

Record each huddle **server-side, with audio bound to each speaker's pubkey**, so it can be transcribed and the transcript lands in the channel log as signed events. Today a huddle's spoken content is the one modality that never becomes part of the event log — so agents watching a channel can see that a huddle happened, but never what was said.

## Current state (as I read the code)

- **The relay only forwards audio; it persists nothing.** `crates/buzz-relay/src/audio/*` mixes/routes live streams. The only huddle events written to the log are lifecycle metadata — `KIND_HUDDLE_STARTED/PARTICIPANT_JOINED/PARTICIPANT_LEFT/ENDED` (`buzz-core/src/kind.rs`). Reaction bursts are explicitly ephemeral and never stored.
- **The relay already knows who is speaking.** The audio room keys every participant by `(pubkey, peer_index)` and maintains a `peer_index → pubkey` roster (`crates/buzz-relay/src/audio/room.rs`). Per-speaker attribution is already inherent server-side — no diarization guesswork needed.
- **Transcription exists, but only client-side and ephemeral.** It runs on-device in the desktop app (`desktop/src-tauri/src/huddle/{stt,transcription,pipeline}.rs`, embedded sherpa-onnx + Parakeet English-only). It is not recorded, not durable, and not written to the log as events. (#2478 tracks the English-only limitation; this request is orthogonal — it is about *where* the recording/transcript live, not *which* STT model runs.)

So the gap: **no durable per-speaker recording, and no transcript in the event log.**

## Why it matters

Buzz's whole premise is "one event log, one search index — humans, agents, workflows all speak it." Huddles break that promise: their content evaporates. The "incident memory" story in the README ("an agent pulls six months of history") simply cannot include anything anyone ever said out loud. Making the transcript a first-class signed event closes the last modality gap, and it does so for agents specifically — an agent that is a channel member would get huddle content as context the same way it gets messages today.

## Proposal

1. **Server-side recording, per speaker.** Persist each huddle's audio keyed by speaker pubkey (the relay already has `(pubkey, peer_index)`), to the existing media store (Blossom/S3). Per-speaker streams make diarization free and let transcription attribute every segment to an npub.
2. **Transcription against the per-speaker audio.** Run STT server-side (or via a pluggable/off-box transcriber) over each speaker's track. Because segments are already attributed, output is "speaker → text → time range" with no speaker-separation heuristics.
3. **Transcript as signed events in the channel log.** Emit transcript segments as signed events scoped to the huddle + channel, tagged with speaker pubkey and a timestamp range — searchable via the existing FTS index, and readable by agents as ordinary channel context. A dedicated kind (or an extension of the huddle-lifecycle range) keeps them identifiable without decrypting.

This composes cleanly with #2451 (evidence-backed knowledge crystallization as channel-scoped signed events): a huddle transcript is exactly the kind of evidence that RFC wants to crystallize.

## Notes / considerations

- **Consent is not optional.** Recording a conversation is sensitive; it should be opt-in per huddle and visibly announced to participants. The existing `KIND_HUDDLE_GUIDELINES` (48106) is a natural place to carry a "this huddle is recorded" policy.
- **Storage/retention.** Media store already exists (S3/MinIO/Blossom); recordings need a retention policy and a way to redact/delete (a NIP-09-style deletion for the transcript events + object cleanup).
- **Pluggable STT.** Given #2478, the transcriber should be swappable and language-aware rather than pinned. A per-speaker pipeline also lets different speakers use different language models where locale is known.

Happy to help — we run a Parakeet-based transcription pipeline elsewhere and would gladly test a per-speaker, server-side path (including non-English, which ties back to #2478).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.