cockroachdb / cockroachdb/cockroach

backup: elevated tail latencies in SQL workload while backing up to s3

Open
#115,190 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug O-support P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We have observed that backups to s3 cause increased tail latencies in foreground traffic, sometimes significantly.

We currently see some cases where 600+ of heap is inuse by the chunk buffers in the sdk, leading to increased rates of GC (even absent memory pressure but rather just due to its size relative to the live heap if a reasonable GOGC is not set e.g. due to #[115164](https://github.com/cockroachdb/cockroach/issues/115164) ).

These more frequent GC runs appear to also see higher per-run pause times, sometimes much higher.

The S3 SDK hashes chunk sized (currently 8mb) blocks both with MD5 and SHA256, for content checksum and signing respectively. It appears that due in large part to https://github.com/golang/go/issues/64417, this causes us to observe long gc pause times and traces show STW pauses overlapping with block hashing.

This is a tracking issue for all related issues.
- [x] https://github.com/cockroachdb/cockroach/issues/115194
- [x] https://github.com/cockroachdb/cockroach/issues/115189
- [x] https://github.com/cockroachdb/cockroach/issues/115192
- [x] https://github.com/cockroachdb/cockroach/issues/115164
- [x] https://github.com/cockroachdb/cockroach/issues/115193
- [ ] https://github.com/cockroachdb/cockroach/issues/115196

Jira issue: CRDB-33924

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.