block / block/buzz

Archive observer telemetry bypasses the live 3000-event cap; storage grows 15 MB/day

Open
#5,985 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

**Describe the bug**
Two related but separately severable problems, found from opposite ends —
one by measuring a live installation, one by reading the Desktop sources.

**High — the archive path is unbounded.** The live observer path is capped at
`MAX_OBSERVER_EVENTS = 3000`, but the channel-scoped archive read from SQLite
has no equivalent bound, and the UI merges the two before building the
transcript. Long-lived agent channels therefore become progressively more
expensive to open and render, and the existing cap does not prevent it.

**Medium — retention and storage efficiency.** Archived `kind 24200` telemetry
is persisted indefinitely and almost never becomes a durable metric record:
0.76% conversion on the installation measured. Disk and CPU grow without
proportional product value.

Chat messages are not involved — there are none in the database at all.

The user-visible symptom is the reasoning/transcript window lagging in a long
thread on otherwise idle high-end hardware.

**Steps to reproduce**
1. Run an agent in a channel for a day or two so telemetry accumulates
(~3000 events/day observed with a single active agent).
2. Open that channel in Desktop with the reasoning/transcript window visible.
3. Observe input and scrolling lag, growing with the age of the channel.

Confirm the underlying numbers directly against the local database:

```bash
# share of telemetry in the local database
sqlite3 ~/.buzz/archive/archive.db \
"select kind, count(*), sum(length(raw_json))/1024/1024 from archived_events group by kind;"

# accumulation rate
sqlite3 ~/.buzz/archive/archive.db \
"select count(*), (max(archived_at)-min(archived_at))/86400.0 from archived_events;"

# conversion rate into metrics
sqlite3 ~/.buzz/archive/archive.db \
"select (select count(*) from agent_metric_index)*100.0/(select count(*) from archived_events);"

# verify the index is used (expect COVERING INDEX, not SCAN)
sqlite3 ~/.buzz/archive/archive.db \
"explain query plan select id from observer_channel_index
where identity_pubkey=? and relay_url=? and channel_id=?
order by created_at desc, id desc limit 100;"

**Expected behavior**
Opening a channel should cost roughly the same regardless of how long an agent
has been working in it. The MAX_OBSERVER_EVENTS = 3000 bound that protects
the live path should also bound what the merge step has to process, and
telemetry that never becomes a metric should not be retained indefinitely.

Instead, the archive read is unbounded, so channel open time scales with
channel age, and the database grows without a ceiling.

**Version and platform**

- Buzz version: 0.5.14 (buzz-desktop.exe, 91 MB, single native process)
- OS: Windows 11 Pro, build 26100
- CPU: AMD Ryzen 7 7700, 8C/16T
- RAM: 63 GB total, 33 GB free
- Relay: buzz.supermvp.space

Idle footprint is healthy: 127 MB RSS and 0.3% of one core. The slowdown
appears only on a channel where an agent has been working.

**Logs / additional context**
Measurements

Contents of archived_events, grouped by kind (33.4 MB database after 1.43 days):

kind events bytes share
─────── ──────── ─────────── ──────
24200 4308 23 323 328 99.9% agent telemetry
44200 33 17 603 0.1%
─────── ──────── ─────────── ──────
TOTAL 4341 23 340 931 22.26 MB

99.9% of stored volume is agent telemetry. Kind 9 (chat messages) does not
appear in the database at all.

Growth rate over the measured span:

3026 events/day
15.5 MB/day

Projected size of the database file (currently 33.4 MB):

after 7 days ~142 MB
after 30 days ~498 MB
after 90 days ~1.4 GB
after 365 days ~5.7 GB

Per-channel breakdown for one active channel, all events from a single agent:

date events volume
─────── ─────── ────────
08-14 2108 11.2 MB
08-15 405 2.8 MB (partial day)

Storage efficiency

Telemetry is stored in full but almost never consumed:

telemetry events stored 4341
rows in agent_metric_index 33 (all parse_status = valid)
conversion rate 0.76%

99.24% of the stored bytes never become a metric. Because the payloads are
encrypted (frame: telemetry), each one also has to be decrypted when
processed — the cost is paid in CPU as well as in disk.

event size count volume
─────────── ───── ───────
0-2 KB 1383 1.8 MB
2-8 KB 2320 8.5 MB
8-32 KB 375 5.6 MB
32+ KB 118 5.8 MB (largest single event: 86 KB)

Root cause

High — the archive path is not bounded. Inspection of the Desktop sources
(block/buzz at 6e0631f6b5d2139e4e080bf94e27ecee8a3d4d74):

MAX_OBSERVER_EVENTS = 3000 desktop/src/features/agents/observerRelayStore.ts:29
live path truncates to 3000 observerRelayStore.ts:208-212
archive read has no cap observerRelayStore.ts:52-58
merge before transcript ManagedAgentSessionPanel.tsx:89-93

The measurements confirm this is reached in practice, not just in theory. The
channel examined here already holds 2607 archived events and, at the observed
rate, crosses the 3000 live cap within roughly three hours. The bound that
protects the live path does not protect the archive path.

Medium — the subscription has no retention bound.

save_subscriptions: scope_type=owner_p, kinds=[24200, 44200]

The owner is subscribed to telemetry from all of their agents, and every event
is persisted indefinitely. On its own this is a storage and CPU cost rather
than a rendering defect — but it is what supplies the unbounded archive above
with material.

What was ruled out

Hypothesis Check
──────────────────────────────── ──────────────────────────────────────
Insufficient RAM or CPU 63 GB RAM with 33 GB free; 0.3% of one
core at idle

Missing index on channel idx_observer_channel exists and is used:
100 events in 0.1 ms when queried with
the full key (identity_pubkey +
relay_url + channel_id). Omitting the
leading columns causes a full SCAN and
can be mistaken for a missing index.

Heavy attachment rendering 8 images across 34 messages — wrong
order of magnitude

Proposed fixes

High — bound the archive path:

Change Effect
────────────────────────────────── ──────────────────────────────────
Apply a bound to the archive read the merge step stops scaling with
path equivalent to channel age; the existing cap
MAX_OBSERVER_EVENTS, or paginate starts actually holding
the merge

Do not load kind 24200 when channel open time stops scaling
opening a channel — telemetry with agent activity
belongs to the metrics panel,
not the chat window

Medium — retention and efficiency:

Change Effect
────────────────────────────────── ──────────────────────────────────
Retention policy for telemetry removes ~99% of DB volume
(N days or M events), enforced
on startup

Parse into a metric on arrival removes 99.24% of stored volume at
and drop raw_json afterwards identical metric completeness

VACUUM after cleanup file size actually shrinks

Low — unrelated waste found along the way:

Change Effect
────────────────────────────────── ──────────────────────────────────
Send UTF-8 instead of \uXXXX for ~55% less traffic on non-Latin
thread payloads threads (see below)

Secondary finding: JSON escaping

Cyrillic content is escaped as \uXXXX in transit, costing 6 bytes per
character. A 34-message thread transfers as 276 KB instead of 125 KB — a 2.3×
inflation. This does not explain the lag, but it is measurable waste on any
non-Latin channel.

Scope of these claims

Measurements were taken from the local database on one Windows machine. The
root cause is inferred from that data plus the source references above; it has
not been confirmed by reading the persistence code itself. Anyone with access
to the Desktop sources can confirm or refute it quickly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.