apache / apache/iggy

Disk polling: investigate tail latency under concurrent writes

Open
#4,178 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Part of #4174, which records the compared revisions, workload and statistical method.

Investigate the unresolved tail behavior under concurrent writes. Start with archived timelines; collect more diagnostics only when a recurring delay or an explicit tail requirement makes the decision useful.

| Case | Change in run p99 | Pointwise 95% interval |
| --- | ---: | --- |
| Explicit offset | −0.02% | [−2.79%, +2.88%] |
| Explicit offset with writes | +3.40% | [−0.62%, +7.63%] |
| Next | −0.84% | [−2.37%, +0.69%] |
| Consumer group | +0.86% | [−0.89%, +2.65%] |
| Resident | −2.33% | [−6.29%, +1.23%] |

Positive values mean higher p99. Estimates are geometric means of paired ratios of run p99 values, not pooled request percentiles. They use the same pair bootstrap described in #4174. The interval with writes includes a small improvement and a meaningful increase, so it neither confirms a tail regression nor resolves a 3% margin. Mean elapsed time is a separate metric, reported in the parent.

The preceding campaign with the same binaries reported Next p99 +6.79% and group p99 +6.06%; those increases were not reproduced. Keep the campaigns separate. The rerun followed reported high Docker memory consumption and a guest restart, but memory pressure was not independently controlled and continuously measured. No container CPU throttling or memory limit events were recorded; those checks cannot exclude host pressure or establish a memory cause.

1. Inspect the [published raw timelines and resource records](https://github.com/diegomrsantos/iggy/releases/tag/pr-4119-final-review-evidence-2026-09-13), including run order and valid slow operations. Look for clustering around a stage or resource condition.
2. Reuse the parent's diagnostics and timestamps from #4177. Add sparse stage timestamps, scheduler waits or queue observations only where needed to locate the delay. CPU profiles alone cannot locate waiting.
3. State the latency boundary, load model and worthwhile effect before fresh confirmation. Follow the parent's fixed budget and arrival model rules, including scheduling delays, errors and timeouts.

Tracing can create the tail being studied. Preserve valid slow runs and stop according to the declared budget. Repeating measurements on the same VM cannot remove systematic environment bias or establish physical device latency. A change should address a supported mechanism, not merely a difference between campaigns.

Contributor guide

Open the contributing guide

Research direction

Start with the published raw timelines and resource records from the linked release, keeping run order and valid slow operations separate. Reuse the diagnostics and timestamps from #4177, adding sparse stage or queue observations only when needed. Done means stating the latency boundary and load model, identifying a supported mechanism if one exists, and following the parent issue's fixed budget and arrival-model rules.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.