0xMiden / 0xMiden/note-transport-service
StreamNotes resource/DoS hardening (subscriptions, polling, cloning, leaks)
- Langage dominant
- Rust
- Étoiles
- 3
- Forks
- 10
- Merge moyen
- 2 h 23 min
- PR mergées (30 j)
- 4
Description
Severity: high/medium.
### Summary
The streaming subsystem has several independent resource issues; grouping them since they touch the same code and should be fixed together.
### Findings
1. **Unbounded subscriptions.** `stream_notes` registers a sub unconditionally (`crates/node/src/node/grpc/mod.rs:291-311`); no per-connection or global cap. The `GlobalConcurrencyLimitLayer` permit is released once the *response future* resolves (stream established), so it does not bound live streams.
2. **O(distinct-tags) DB polling per tick.** `streaming.rs:96-97` issues one `fetch_notes` per tracked tag, sequentially, every 500 ms, against single-writer SQLite. Subscribing to many distinct tags stretches the tick arbitrarily. The multi-tag `fetch_notes_by_tags` already exists and should be used.
3. **Control-message starvation.** `streaming.rs:220-237` races `query_updates()` (which begins with `sleep(500ms)`, line 93) against `rx.recv()` in a `select!`; every control message (including the per-poll `Waker`, line 287) cancels and restarts the 500 ms timer. Enough control traffic and the DB poll never runs. Use a persistent `tokio::time::interval` outside the select.
4. **Per-subscriber batch cloning.** `streaming.rs:130` does `notes.clone()` per subscriber (up to ~256 MB), so N subscribers on a hot tag multiply memory; each sub's channel buffers up to 32 batches (`mpsc::channel(32)`, `grpc/mod.rs:304`). Share via `Arc`, shrink the buffer.
5. **Subscription leaks / id collisions.** `Sub::drop` uses `try_send` on a capacity-128 control channel (`streaming.rs:297-310`); if full, the RemoveSub is lost and the tag stays polled forever. Subscription ids are `rand::random::()` with no collision check (`grpc/mod.rs:297`), so a collision overwrites another client's entry. Use an atomic counter and a non-lossy control channel.
6. **Full-backlog replay on resubscribe.** When the last sub for a tag drops, `TagData` (and its cursor) is removed (`streaming.rs:175-179`); a new sub recreates it at `cursor: 0` (`streaming.rs:262`), replaying the whole 30-day backlog 500 rows/tick. Cheap amplification loop (connect, drain, disconnect, repeat). Fixed together with per-subscriber cursors (#96).
### Recommendation
Per-connection + global subscription caps; one batched `fetch_notes_by_tags` per tick; a fixed interval decoupled from control handling; `Arc`-shared batches with a small buffer; atomic ids + non-lossy lifecycle channel; per-subscriber cursors.
Related: #96, #28 (client reconnects).
---
Part of #114.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Les principaux fichiers sont `crates/node/src/node/grpc/mod.rs` et `streaming.rs`. Commencez par comprendre le flux d'abonnement et le cycle de vie de `TagData`. Examinez les fonctions `fetch_notes` et `fetch_notes_by_tags`, la gestion du canal de contrôle et le clonage par lots. Les tests impliquent de simuler plusieurs abonnements et d'observer le comportement de la mémoire et du sondage. 'Terminé' signifie que les six constatations sont traitées sans compromettre la fonctionnalité de flux existante.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- sqlite
- Domaine
- backend, databases, performance
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 45/100