cockroachdb / cockroachdb/cockroach
backup: export failed errors on retry are too noisy
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
On `drt-scale` During incremental backup there are many failed export request error message. The [incremental backup](https://drt-scale-0012.drt.crdb.io:26258/#/jobs/1018767606024831048) did complete successfully in 6 min. Logs link on [datadog](https://us5.datadoghq.com/logs?query=status%3Aerror%20cluster%3Adrt-scale%20-%22failed%20to%20replicate%20after%205%20retries%22%20-%22cannot%20process%20destroyed%20replica%22%20-%22TLS%20handshake%22%20-%22checksum%20computation%20failed%22%20-%22background%20error%3A%20pebble%3A%20compaction%20cancelled%20by%20a%20concurrent%20operation%2C%20will%20retry%20compaction%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=time%2Casc&viz=stream&from_ts=1730973499490&to_ts=1730973994310&live=false).
```
failed export request ‹/Table/113/4/151{397/76453-404/56910}›
trace:
0.000ms 0.000ms === operation:backupccl.ExportRequest
0.000ms 0.000ms [dist sender send: {count: 1, duration 491ms}]
0.000ms 0.000ms [/cockroach.roachpb.Internal/Batch: {count: 1, duration 491ms}]
0.000ms 0.000ms [admissionWorkQueueWait: {count: 1, duration 491ms}]
490.651ms 490.651ms structured:‹{"@type":"type.googleapis.com/cockroach.util.admission.admissionpb.AdmissionWorkQueueStats","waitDurationNanos":"0.490573770s","queueKind":"kv-elastic-cpu-queue","deadlineExceeded":true,"workPriority":"bulk-normal-pri"}›
```
This issue is request to change log severity from error to warning. Here is the [code](https://github.com/cockroachdb/cockroach/blob/release-24.3/pkg/ccl/backupccl/backup_processor.go#L613) which needs to be changed.
More details in [thread](https://cockroachlabs.slack.com/archives/C07HPMBLVJ7/p1730966038418579).
Jira issue: CRDB-44234
Contributor guide
Assessment
This issue has not been assessed yet.