apache / apache/iggy

Disk polling: investigate repeated batch verification and record scanning

Open
#4,176 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Part of #4174, which records the shared benchmark setup, revisions and evaluation criteria.

[walk_disk_chunk](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/partitions/src/poll_plan.rs) decodes a complete batch and verifies checksums when enabled. [select_batch_slice](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/partitions/src/journal.rs) then walks records to locate the requested slice. Adjacent small polls can revisit the same batch, and selecting a late record can scan preceding records.

Historical profiles measured substantial CPU time in the inclusive checksum and hash category:

| Workload | Baseline sampled CPU, µs per poll | Candidate sampled CPU, µs per poll |
| --- | ---: | ---: |
| Explicit offset, without concurrent writes | 100.474 | 101.810 |
| Explicit offset, with concurrent writes | 109.583 | 110.524 |

The category includes called functions and overlaps its callers, so it cannot be added to their CPU totals. These samples establish neither repeated verification nor avoidable work. The shared sampling method and workload limits are in #4174.

[verify_and_recompute_batch_checksum](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/binary_protocol/src/batch.rs) already validates message bodies and recomputes the batch checksum in one frame walk. Investigate repeated visits and selection costs before assuming a duplicate pass exists.

1. Count bytes hashed, records visited, distinct batches and repeat visits per returned message. Attribute current hash samples to callers and distinguish verification from record selection.
2. Vary batch size, requested message count and position within a batch. Determine whether the cost comes from large batches, repeated visits, locating a late record or the hash implementation.
3. Share counters with #4175, then test one verification or parsing change separately from read sizing. Confirm any component saving through polling, including larger requests and concurrent writes.

Preserve configured checksum and body validation, malformed frame rejection and the parent's output guarantees. A previously verified file range does not establish the integrity of newly read bytes: corruption and path reuse can occur later. Any retained verified buffer needs bounded memory, correct identity and explicit invalidation.

Contributor guide

Open the contributing guide

Research direction

Start with walk_disk_chunk in core/partitions/src/poll_plan.rs, select_batch_slice in core/partitions/src/journal.rs, and verify_and_recompute_batch_checksum in core/binary_protocol/src/batch.rs. Measure bytes hashed, records visited, distinct batches, repeat visits, and selection position across the workloads in #4174, sharing counters with #4175. Done means the costs are attributed, one verification or parsing change is tested separately from read sizing, and checksum, validation, malformed-frame rejection, output, memory, identity, and invalidation guarantees remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.