cockroachdb / cockroachdb/cockroach
kvserver: send download span requests through DistSender/Raft
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
During the download phase of Online Restore, the job coordinator [pokes](https://github.com/jeffswenson/cockroach/blob/jeffswenson-sst-extract-batch/pkg/backup/restore_online.go#L806) all stores to download the newly restored external keyspace, [via the status server](https://github.com/jeffswenson/cockroach/blob/jeffswenson-sst-extract-batch/pkg/backup/restore_online.go#L590) and [polls every node ](https://github.com/msbutler/cockroach/blob/butler-ldr-table-priv/pkg/backup/restore_online.go#L490) to collect replica level ExternalBytes counts. So ideally, as long as a key span is a part of a replica with External Bytes, we will re-issue Download Span requests. In other words, the Download job _should_ not exit until all replicas for the relevant key space have 0 external bytes.
This quick and dirty solution has 2 downsides:
1. The status server infra sends the request to _all_ stores. We only need to send the request to stores that actually contain the relevant external ssts.
2. Our external bytes poller can exit even if there are still external bytes, as it is unaware of changing replica sets. Consider this example with a stalled external file snapshot:
- t1: send a snapshot with virtual ssts from n1 to n2, but writing the external ssts onto n2 stalls.
- t2: all virtual ssts on n1 are downloaded
- t3: poller detects 0 external bytes in the cluster, even though theres a pending snapshot from n1 to n2 that will write external ssts to n2
- we can construct another example that involves temporarily decomissioning a node with external data
We do not believe the race described in 2) is that big of a deal: If the download job completes before the snapshot receiver downloads the external sst and the backing file gets deleted, we do not lose quorum, as there must exist two replicas that already downloaded the file. So, a blunt approach to handle this race would be to kill the node with the borked file. But, it would be nice to guarantee that once the download job completes, the cluster does not contain any external data.
To solve these 2 problems, we ought to send download span requests through a normal kv api that leverages distsender and raft to atomically download the key span on the relevant replica set.
One more note here: when we recover from a failed or cancelled online restore, we [scrub](https://github.com/cockroachdb/cockroach/pull/143334) all external data via excise span requests that reach all replicas through raft.
UPDATE: solving problem 1 with distsender may not be sufficient, as we actually need to send the download span request to all relevant _stores_, not replicas. As an example, there may be node that used be in the replica set that has external bytes in a given span.
Jira issue: CRDB-52281
Contributor guide
Research direction
Read pkg/backup/restore_online.go at the coordinator, status-server, and external-bytes polling entry points linked in the issue, then compare the existing excise-span-through-Raft path referenced by PR 143334. Done means download-span requests reach the relevant stores through the normal KV path and completion guarantees that no relevant replicas or former stores retain external data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100