Disk polling: performance investigation and optimization tracking

Track the performance investigations following #4119, starting with excess read bytes and repeated batch processing. Owner service, tail latency and task construction need fresh diagnostics before choosing an optimization.

- [ ] #4175 - Excess read bytes for small responses
- [ ] #4176 - Repeated batch verification and record scanning
- [ ] #4177 - Owner completion service under concurrent polling
- [ ] #4178 - Tail latency under concurrent writes
- [ ] #4179 - Task construction and completion payload costs

Read sizing and batch processing can share counters. Owner service and tail latency can share stage timestamps. Each issue should reach its own measurement and decision; finding that an optimization is not worthwhile is a valid outcome.

The [latest report and raw evidence](https://github.com/diegomrsantos/iggy/releases/tag/pr-4119-final-review-evidence-2026-09-13) found higher mean poll time with concurrent writes and lower mean time for resident reads. The other mean intervals span zero, which does not establish equivalence.

| Case | Baseline µs per poll | Candidate µs per poll | Mean time change | Pointwise 95% interval |
| --- | ---: | ---: | ---: | --- |
| Explicit offset | 258.42 | 258.67 | +0.10% | [−1.35%, +1.69%] |
| Explicit offset with writes | 291.93 | 298.71 | +2.32% | [+0.41%, +4.36%] |
| Next | 303.86 | 302.40 | −0.48% | [−1.41%, +0.44%] |
| Consumer group | 330.16 | 331.63 | +0.45% | [−0.59%, +1.45%] |
| Resident | 133.44 | 128.41 | −3.77% | [−5.27%, −2.38%] |

Positive changes mean more elapsed time per completed poll. The comparison covers the whole candidate and does not identify the cause of the difference under writes. Tail results and differences between campaigns are in #4178. Historical copy, checksum and task construction evidence is retained beside the relevant investigation in #4175, #4176 and #4179.

Start by inspecting the archived run, resource and producer records, then collect a small fixed diagnostic set on current code with and without writes. Share counts of bytes read, bytes hashed, batches visited and pump turns, together with:

read start → completion → enqueue → owner acceptance → caller resumption

For each proposed change, compare one revision with the same revision plus only that change. Keep instrumentation separate from clean timing, record exact binary identities, validate returned work, and measure elapsed time, CPU and memory. Assess variation with identical binaries where needed and declare a worthwhile effect and fixed budget before confirmation. Coordinate representative Linux workloads with existing benchmark work.

Preserve output, integrity, durability and owner acceptance contracts, including offsets, timestamps, contiguous results and consumer progress. For independent arrival workloads, account for scheduling delays, errors and timeouts: a sequential client reduces its load when polling slows. Keep write progress and other poll modes as guardrails.

The tracker closes when every child has a documented result and decision or is explicitly removed from scope.

Shared evidence, methods and workload details

**Latest timing campaign**

Baseline `585ab692210b4e900cdc1153e75842378a33370e`; candidate `c7e6fb1655bd48c18c2fb567917f88ec1bdf7fcc`. Twelve independently launched pairs per case used balanced randomized order. All 120 measured runs completed 2,304,000 nonempty polls without recorded errors, timeouts or cancellations. The separate 20 run pilot is excluded.

Mean estimates are ratios of arithmetic means. Pointwise percentile intervals use 20,000 bootstrap resamples of whole pairs, without adjustment across cases or secondary metrics. The fixed sample was not extended according to its results.

**Historical CPU profiles used in #4175, #4176 and #4179**

Baseline `dc2b382097ed7fb6cfeb8a31fed0fbc74af4433e`; candidate `a699ff91c6b9126502ea112ac676d96d8d107975`. Two captures per revision and workload each completed 33,000 polls, with opposite version orders. Linux perf sampled software `cpu-clock` at 499 Hz; sample periods were normalized by completed polls. Tables report arithmetic means across captures.

These are diagnostic CPU estimates, not elapsed latency or predicted savings. Profiling can perturb execution, and no confidence intervals are assigned to this small set. The detailed reports and raw captures remain archived locally; relevant observations are reproduced in the children. They are separate from the published timing archive. #4177 records a different diagnostic with its own revision and sample size.

**Workloads and environment**

All these workloads used one shard, one sequential TCP consumer and payloads of 256 bytes. Explicit offset polls returned one message without automatic commits. In the latest campaign, Next and resident polls returned one message with automatic commits; group polls returned 100. Explicit offset runs measured 33,000 polls, and the other latest cases measured 10,000. Separate fixtures supplied 100 warmup polls. Preparation, preload and shutdown were outside the consumer window.

The producer targeted 50,000,000 payload bytes per second in batches of 100. In the latest campaign it achieved 50,154,102.84 to 50,505,824.75 bytes per second, within the 95–105% guard. Written messages per completed poll averaged 57.45 for baseline and 58.81 for candidate. Mixed server CPU includes serving writes and is not consumer CPU alone.

Measurements ran in Docker Desktop's ARM64 Linux VM on an M1 Pro MacBookPro18,1, with server work pinned to CPU 0 and fixtures on a guest volume. Reads could hit the OS cache. The latest disk cases used replicated durability, a flush threshold of one, 16 MiB segments and disabled preallocation. These results do not establish cold storage latency, physical disk traffic, multishard scaling or behavior under many concurrent consumers.

Exact offsets were validated for explicit offset cases. Raw Next and group records do not establish exact returned offsets or payload identity; correctness tests provide separate evidence.

**Source scope**

Child source links are pinned to merged PR head `8f1c0e260076f9fbe49539e904e43895bc24c3fe`. Shutdown receive reuse, reserved completion capacity, numeric history IDs and ownership simplifications are already present there. Historical profiles do not measure their remaining costs.

Start with the archived run, resource and producer records, then read child investigations #4175–#4179 and the current benchmark work. Collect the specified counters and stage timestamps with and without writes, preserving the listed correctness contracts; done means every child has a documented result and decision or is explicitly removed from scope.

Written by the indexing model from the issue text.