juspay / juspay/deja

Profile the record-path stall before landing the sink fixes

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2
Forks
1
Avg merge
21h 32m
Merged PRs (30d)
67

Description

Before landing the recording-sink fixes, attribute the record-path stall with a profiling run. The blocking-flush mechanism explains the saturated regime (concurrency 10, ~518 ms p50), but the unsaturated regime (concurrency 2) still shows ~90 ms/request of record overhead that channel-fill arithmetic does not explain — the bounded channel would need ~18 s of continuous writer stall to fill at that request rate.

## Profiling plan

- A **null-sink control arm** (capture + serialize, no Kafka) to split capture cost from producer interaction.
- Wall-clock attribution across the three candidate blocking points: the flush-control channel send, the writer's producer drain, and serialization.
- Sample the record channel occupancy (already exposed in the writer stats snapshot), count normal-finalize vs drop-path finalize per arm, and histogram producer-flush durations.
- Run the low-concurrency arm from **fresh broker state** to rule out saturation carry-over from a prior run.

## Decision rule

The attribution decides which sink fix (or combination) actually lands: if the flush-control send dominates, the per-request-flush change is sufficient; if the writer's producer drain also parks, add the flush-cadence change; if serialization dominates with high CPU, re-scope to a serialize-cost fix.

Also covers an open measurement caveat: the per-boundary micro-benchmarks (idle ~2 ns, skip ~11 ns) were single-threaded and uncontended — quantify the contended cost under parallel traffic.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the recording sink, writer, and existing writer stats snapshot, then run the null-sink and Kafka profiling arms from fresh broker state. Measure the listed blocking points, channel occupancy, finalize paths, flush durations, and contended boundary cost; done means the measurements identify whether the flush send, producer drain, or serialization determines the sink fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, rust
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.