cockroachdb / cockroachdb/cockroach
Planhook of detached BACKUP repeatedly auto-retries in clusters with large number of descriptors
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We've seen a behavior during backup for a cluster that has a large number descriptors (order of 100k), the planhook would take a large amount of CPU and wall time to deserialize the descriptors (1-2 minutes), and would eventually fail at the end in `CreateAdoptableJobInTxn`, causing the entire planhook to be retried. The row in the `crdb_internal.cluster_transactions` table indicates a last_auto_retry_reason
```
"CreateAdoptableJobInTxn: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "
```
It's currently unclear why CreateAdoptableJobInTxn repeatedly fails with TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): But when the planhook gets to this point it has already been minutes,. I've also been able to sometimes reproduce this same error locally by just inserting a 10s sleep in `LoadAllDescs` and backing up into userfile.
Related internal support Issue: https://github.com/cockroachlabs/support/issues/2367
Jira issue: CRDB-28916
Contributor guide
Assessment
This issue has not been assessed yet.