cockroachdb / cockroachdb/pebble

record: experiment with moving WAL chunk CRC computation to the flush goroutine

Open
#4,431 3 comments 0 reactions 0 assignees View on GitHub
A-storage C-performance T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

Currently, the CRC for each WAL chunk (a.k.a fragment) is computed as the chunk is emitted to a WAL block. This CRC computation is done while holding `commitPipeline.mu` which periodically shows up in mutex profiles. Prior profiling indicates that the CRC computation is ~1/3 of the CPU during `commitPipeline.prepare`. We could move the CRC computation out of `commitPipeline.mu` by not performing it during `LogWriter.emitFragment*` and instead perform it somewhere within `LogWriter.flushLoop`. Before writing a WAL block, or partial WAL block, to the WAL file we'd iterate over the fragments being written and populate the CRC. This iteration is straightforward as the fragments are tightly packed in the WAL block and are self-describing:

```
+----------+-----------+-----------+----------------+--- ... ---+----------+-----+
| CRC (4B) | Size (2B) | Type (1B) | Log number (4B)| Payload | CRC (4B) | ... |
+----------+-----------+-----------+----------------+--- ... ---+----------+-----+
```

It isn't clear that this refactoring will be a win as performing the CRC computation on the flush goroutine will compete with time spent performing I/O. A quick experiment to see if this may be worthwhile would be to benchmark disabling the CRC computation in `LogWriter.emitFragment*`.

Jira issue: PEBBLE-368

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.