basalt-26.3: roachtest: backup-restore/chaos failed

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.backup-restore/chaos [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21491316?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21491316?buildTab=artifacts#/backup-restore/chaos) on basalt-26.3 @ [23c3d5a11cfe60e3d3650467f2be08b000eba5ef](https://github.com/cockroachdb/cockroach/commits/23c3d5a11cfe60e3d3650467f2be08b000eba5ef):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:468
pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:377
pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:133
pkg/cmd/roachtest/test_runner.go:1558
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
retries exhausted waiting for job 1205589425468964867 to pass 76.26% progress
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*CommonTestUtils).waitForJobFractionCompletedWithNode
| pkg/cmd/roachtest/tests/backup_restore_utils.go:603
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.injectAndRecoverFailure.func1
| pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:443
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) retries exhausted waiting for job 1205589425468964867 to pass 76.26% progress
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: backup-restore/chaos
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/backup-restore/chaos/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21491316-1787983402-32-n4cpu4-0001 | | 10.91.99.8 |
| teamcity-21491316-1787983402-32-n4cpu4-0002 | | 10.91.99.9 |
| teamcity-21491316-1787983402-32-n4cpu4-0003 | | 10.91.99.5 |
| teamcity-21491316-1787983402-32-n4cpu4-0004 | | 10.91.99.13 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicDRPC=true
- onlineRestore=false
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21491316/backup-restore-chaos/1787984967496/1787992698977)

_Logs were not uploaded to Datadog_

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*backup-restore/chaos.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start with the backup-restore/chaos artifacts and logs, then trace the failure through pkg/cmd/roachtest/tests/backup_restore_roundtrip.go and backup_restore_utils.go, especially waitForJobFractionCompletedWithNode. Compare the failing run with the referenced commit and investigate why job 1205589425468964867 cannot reach 76.26% progress. Done means identifying and fixing the failure, with the backup-restore/chaos roachtest passing reliably.

Written by the indexing model from the issue text.