cockroachdb / cockroachdb/cockroach
importer: `ExternalStorage.List` call should be moved out of planning into job execution
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In https://github.com/cockroachlabs/support/issues/2375 we saw an instance of an import taking several minutes in planning because of the number of objects import had to list during wildcard expansion. We hold a txn open for this duration on the jobs table that was frequently refreshed causing the List call to re-resume. The txn also failed to commit multiple times with https://www.cockroachlabs.com/docs/v22.2/transaction-retry-error-reference.html#abort_reason_client_reject.
Planning should be cheap and not reach out to ExternalStorage. We should move this step into job execution where we are not holding a txn open on the jobs table.
Jira issue: CRDB-28696
Contributor guide
Assessment
This issue has not been assessed yet.