pkg/backup/backuprand/backuprand_test: TestBackupRestoreRandomDataRoundtrips failed

pkg/backup/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips [failed](https://mesolite.cluster.engflow.com/invocations/default/6e2f432b-367d-44af-ac0b-d9a71a4ac63d?testReportRun=1&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvYmFja3VwL2JhY2t1cHJhbmQ6YmFja3VwcmFuZF90ZXN0) on release-25.2.24-rc @ [92c7311721160180bed1228023264f6df947fc1e](https://github.com/cockroachdb/cockroach/commits/92c7311721160180bed1228023264f6df947fc1e):

```
github.com/cockroachdb/cockroach/pkg/sql/gcjob.deleteTableDescriptorsAfterGC({0x84b3338, 0xc0192cacf0}, 0xc019e1a008, 0xc0195cff40, 0xc0196a65f8)
pkg/sql/gcjob/table_garbage_collection.go:309 +0x325
github.com/cockroachdb/cockroach/pkg/sql/gcjob.waitForGC({0x84b3338?, 0xc0192cacf0?}, 0xc019e1a001?, 0xc019192000?, 0xc0196a65f8?)
pkg/sql/gcjob/gc_job.go:414 +0x2a
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.deleteDataAndWaitForGC({_}, {_, _}, {0xc003b40000, 0xc002067dd0, {0xc001ae90c0, 0xc00155ff20, 0xc001ae90b0, 0xc003e6b060}, {{0xcf35d00, ...}, ...}, ...}, ...)
pkg/sql/gcjob/gc_job.go:397 +0x185
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume({0xc019192000}, {0x84b3338, 0xc0192cacf0}, {0x713d380?, 0xc01ce5f3b0?})
pkg/sql/gcjob/gc_job.go:376 +0x2ff
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc019192000?, 0x0?, 0xc01de41890, {0x84a3030?, 0xc01ce5f190?}, {0x84b3338?, 0xc0192cacf0?}, {0x713d380?, 0xc01ce5f3b0?})
pkg/jobs/registry.go:1758 +0xac
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc00239b108, {0x84b3300, 0xc019281680}, {0x713d380, 0xc01ce5f3b0}, {0x84a3030, 0xc01ce5f190}, 0xc019192000, {0xc019148600, 0x7}, ...)
pkg/jobs/registry.go:1759 +0xba5
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc00239b108, {0x84b3300, 0xc019281680}, {0x84a3030, 0xc01ce5f190}, 0xc019192000, {0xc019148600, 0x7}, {0xc0114486d8, 0x17})
pkg/jobs/adopt.go:415 +0x545
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0x84b4758?, 0xc0191a7a40?})
pkg/jobs/adopt.go:268 +0x13c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2({0xc00e9599c8, 0x17})
pkg/util/stop/stopper.go:507 +0x1e2
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 23264
pkg/util/stop/stopper.go:496 +0x466

goroutine 8317 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/gcjob.waitForEmptyPrefix({0x84b3338, 0xc010fd2240}, 0xc001b09d60, 0xc003b40000, 0xc0?, 0x0, {0xc016f76eb0, 0x2, 0x8})
pkg/sql/gcjob/gc_job.go:457 +0x2c5
github.com/cockroachdb/cockroach/pkg/sql/gcjob.deleteTableDescriptorsAfterGC({0x84b3338, 0xc010fd2240}, 0xc00a682608, 0xc008a17480, 0xc00e03da98)
pkg/sql/gcjob/table_garbage_collection.go:309 +0x325
github.com/cockroachdb/cockroach/pkg/sql/gcjob.waitForGC({0x84b3338?, 0xc010fd2240?}, 0xc00a682601?, 0xc0036eed80?, 0xc00e03da98?)
pkg/sql/gcjob/gc_job.go:414 +0x2a
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.deleteDataAndWaitForGC({_}, {_, _}, {0xc003b40000, 0xc002067dd0, {0xc001ae90c0, 0xc00155ff20, 0xc001ae90b0, 0xc003e6b060}, {{0xcf35d00, ...}, ...}, ...}, ...)
pkg/sql/gcjob/gc_job.go:397 +0x185
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume({0xc0036eed80}, {0x84b3338, 0xc010fd2240}, {0x713d380?, 0xc0150f55e8?})
pkg/sql/gcjob/gc_job.go:376 +0x2ff
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc0036eed80?, 0x0?, 0xc0000d5890, {0x84a3030?, 0xc0150f5558?}, {0x84b3338?, 0xc010fd2240?}, {0x713d380?, 0xc0150f55e8?})
pkg/jobs/registry.go:1758 +0xac
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc00239b108, {0x84b3300, 0xc00cc98f00}, {0x713d380, 0xc0150f55e8}, {0x84a3030, 0xc0150f5558}, 0xc0036eed80, {0xc014b87580, 0x7}, ...)
pkg/jobs/registry.go:1759 +0xba5
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc00239b108, {0x84b3300, 0xc00cc98f00}, {0x84a3030, 0xc0150f5558}, 0xc0036eed80, {0xc014b87580, 0x7}, {0xc004ef8300, 0x17})
pkg/jobs/adopt.go:415 +0x545
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0x84b4758?, 0xc0155b4960?})
pkg/jobs/adopt.go:268 +0x13c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2({0xc004ef8210, 0x17})
pkg/util/stop/stopper.go:507 +0x1e2
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 8310
pkg/util/stop/stopper.go:496 +0x466

goroutine 42216 [chan receive]:
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).drainWriteQueue(0xc00f1cec08)
external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:900 +0xbd
created by github.com/cockroachdb/pebble/sstable.newColumnarWriter in goroutine 41955
external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:192 +0xb85
```

Parameters:
- attempt=1
- run=1
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestBackupRestoreRandomDataRoundtrips.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start by reproducing pkg/backup/backuprand/backuprand_test.TestBackupRestoreRandomDataRoundtrips with the reported attempt, run, and shard parameters. Trace the failure through pkg/sql/gcjob/table_garbage_collection.go and pkg/sql/gcjob/gc_job.go, especially deleteTableDescriptorsAfterGC and waitForEmptyPrefix. Done means the underlying failure is identified and the reported roundtrip test passes reliably.

Written by the indexing model from the issue text.