juspay / juspay/deja

Recording sink: drop on saturation (never stall a live request) + broker-provisioning reuse

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2
Forks
1
Avg merge
21h 32m
Merged PRs (30d)
67

Description

Under a worst-case 100%-recording load test against a single shared Kafka broker, request latency climbed from ~12 ms to ~518 ms p50 and throughput dropped ~9x, while router CPU stayed **low** — the request threads were blocked, not computing.

The record enqueue itself is non-blocking (bounded channel, try-send, fail-open drop). But the per-request flush control message uses a blocking channel send, and each flush message makes the writer run a synchronous multi-second producer drain. Under a saturated broker the writer stops draining, the control channel fills, and request threads block on the send.

## Work items

1. **Drop path.** Make the per-request flush advisory — a non-blocking try-send, or remove the per-request flush entirely (durability is already covered by the count/timer flush policy) — and reserve the long blocking producer drain for shutdown / end-of-stream only. Recording stays fail-open throughout: on saturation the sink drops and marks loss; it must never stall a live payment request. The exact choice is gated on the profiling attribution (separate issue).

2. **Broker-provisioning reuse.** The recording sink should inherit the application's provisioned Kafka brokers (empty broker list ⇒ inherit, non-empty ⇒ explicit override) while keeping its **own** producer client configured for the durability recording needs: acks=all, idempotence, bounded buffer.

Sharing the client *instance* is rejected: the analytics producer carries none of those guarantees, flush is client-wide (recording's flush would drain the analytics buffer), the failure domains would entangle, and recording's ~10x volume would leak into the analytics queue and could broker-throttle it. Sharing brokers + a common base-client constructor gets the provisioning benefit without those couplings.

## Acceptance

A 100%-record load test against provisioned brokers no longer stalls request threads; on deliberate sink saturation, requests continue (fail-open) and dropped records are counted, not silently lost.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the recording sink's per-request flush control path and the broker-provisioning configuration described in the issue; inspect the profiling attribution before choosing whether to remove or make the flush advisory. Use the 100%-record load test against provisioned brokers to verify that request threads never stall, saturation remains fail-open, and dropped records are counted.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, rust
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.