cockroachdb / cockroachdb/pebble
record: experiment with moving WAL chunk CRC computation to the flush goroutine
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
Currently, the CRC for each WAL chunk (a.k.a fragment) is computed as the chunk is emitted to a WAL block. This CRC computation is done while holding `commitPipeline.mu` which periodically shows up in mutex profiles. Prior profiling indicates that the CRC computation is ~1/3 of the CPU during `commitPipeline.prepare`. We could move the CRC computation out of `commitPipeline.mu` by not performing it during `LogWriter.emitFragment*` and instead perform it somewhere within `LogWriter.flushLoop`. Before writing a WAL block, or partial WAL block, to the WAL file we'd iterate over the fragments being written and populate the CRC. This iteration is straightforward as the fragments are tightly packed in the WAL block and are self-describing:
```
+----------+-----------+-----------+----------------+--- ... ---+----------+-----+
| CRC (4B) | Size (2B) | Type (1B) | Log number (4B)| Payload | CRC (4B) | ... |
+----------+-----------+-----------+----------------+--- ... ---+----------+-----+
```
It isn't clear that this refactoring will be a win as performing the CRC computation on the flush goroutine will compete with time spent performing I/O. A quick experiment to see if this may be worthwhile would be to benchmark disabling the CRC computation in `LogWriter.emitFragment*`.
Jira issue: PEBBLE-368
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.