Recording sink: drop on saturation (never stall a live request) + broker-provisioning reuse
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 67
Description
Under a worst-case 100%-recording load test against a single shared Kafka broker, request latency climbed from ~12 ms to ~518 ms p50 and throughput dropped ~9x, while router CPU stayed **low** — the request threads were blocked, not computing.
The record enqueue itself is non-blocking (bounded channel, try-send, fail-open drop). But the per-request flush control message uses a blocking channel send, and each flush message makes the writer run a synchronous multi-second producer drain. Under a saturated broker the writer stops draining, the control channel fills, and request threads block on the send.
## Work items
1. **Drop path.** Make the per-request flush advisory — a non-blocking try-send, or remove the per-request flush entirely (durability is already covered by the count/timer flush policy) — and reserve the long blocking producer drain for shutdown / end-of-stream only. Recording stays fail-open throughout: on saturation the sink drops and marks loss; it must never stall a live payment request. The exact choice is gated on the profiling attribution (separate issue).
2. **Broker-provisioning reuse.** The recording sink should inherit the application's provisioned Kafka brokers (empty broker list ⇒ inherit, non-empty ⇒ explicit override) while keeping its **own** producer client configured for the durability recording needs: acks=all, idempotence, bounded buffer.
Sharing the client *instance* is rejected: the analytics producer carries none of those guarantees, flush is client-wide (recording's flush would drain the analytics buffer), the failure domains would entangle, and recording's ~10x volume would leak into the analytics queue and could broker-throttle it. Sharing brokers + a common base-client constructor gets the provisioning benefit without those couplings.
## Acceptance
A 100%-record load test against provisioned brokers no longer stalls request threads; on deliberate sink saturation, requests continue (fail-open) and dropped records are counted, not silently lost.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the recording sink's per-request flush control path and the broker-provisioning configuration described in the issue; inspect the profiling attribution before choosing whether to remove or make the flush advisory. Use the 100%-record load test against provisioned brokers to verify that request threads never stall, saturation remains fail-open, and dropped records are counted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, rust
- Domain
- backend, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100