cockroachdb / cockroachdb/cockroach

Planhook of detached BACKUP repeatedly auto-retries in clusters with large number of descriptors

Open
#105,187 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We've seen a behavior during backup for a cluster that has a large number descriptors (order of 100k), the planhook would take a large amount of CPU and wall time to deserialize the descriptors (1-2 minutes), and would eventually fail at the end in `CreateAdoptableJobInTxn`, causing the entire planhook to be retried. The row in the `crdb_internal.cluster_transactions` table indicates a last_auto_retry_reason
```
"CreateAdoptableJobInTxn: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "
```

It's currently unclear why CreateAdoptableJobInTxn repeatedly fails with TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): But when the planhook gets to this point it has already been minutes,. I've also been able to sometimes reproduce this same error locally by just inserting a 10s sleep in `LoadAllDescs` and backing up into userfile.

Related internal support Issue: https://github.com/cockroachlabs/support/issues/2367

Jira issue: CRDB-28916

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.