Profile the record-path stall before landing the sink fixes
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 67
Description
Before landing the recording-sink fixes, attribute the record-path stall with a profiling run. The blocking-flush mechanism explains the saturated regime (concurrency 10, ~518 ms p50), but the unsaturated regime (concurrency 2) still shows ~90 ms/request of record overhead that channel-fill arithmetic does not explain — the bounded channel would need ~18 s of continuous writer stall to fill at that request rate.
## Profiling plan
- A **null-sink control arm** (capture + serialize, no Kafka) to split capture cost from producer interaction.
- Wall-clock attribution across the three candidate blocking points: the flush-control channel send, the writer's producer drain, and serialization.
- Sample the record channel occupancy (already exposed in the writer stats snapshot), count normal-finalize vs drop-path finalize per arm, and histogram producer-flush durations.
- Run the low-concurrency arm from **fresh broker state** to rule out saturation carry-over from a prior run.
## Decision rule
The attribution decides which sink fix (or combination) actually lands: if the flush-control send dominates, the per-request-flush change is sufficient; if the writer's producer drain also parks, add the flush-cadence change; if serialization dominates with high CPU, re-scope to a serialize-cost fix.
Also covers an open measurement caveat: the per-boundary micro-benchmarks (idle ~2 ns, skip ~11 ns) were single-threaded and uncontended — quantify the contended cost under parallel traffic.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the recording sink, writer, and existing writer stats snapshot, then run the null-sink and Kafka profiling arms from fresh broker state. Measure the listed blocking points, channel occupancy, finalize paths, flush durations, and contended boundary cost; done means the measurements identify whether the flush send, producer drain, or serialization determines the sink fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, rust
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100