cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: backup/schedule-chaining/tpcc-10 failed

Open
#175,571 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-release-26.3 C-test-failure O-roachtest O-robot release-blocker T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.backup/schedule-chaining/tpcc-10 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21520622?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21520622?buildTab=artifacts#/backup/schedule-chaining/tpcc-10) on release-26.3 @ [7b603fa58c241fb38278b92cfed76c4ed8f34ceb](https://github.com/cockroachdb/cockroach/commits/7b603fa58c241fb38278b92cfed76c4ed8f34ceb):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/backup_chaining.go:227
pkg/cmd/roachtest/test_runner.go:1558
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
backup jobs failed: [{1210965887987908610 failed backup exhausted retries in initial state: running distributed backup to export 27 ranges: exporting /Tenant/3/Table/114/1/{2/50001-4/51707}: batch timestamp 1789627730.697592000,0 must be after replica GC threshold 1789627744.481029835,0 (r127: /Tenant/3/Table/114/1/{2/50001-4/51707})}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*backupDriver).monitorBackups
| pkg/cmd/roachtest/tests/backup_fixtures.go:394
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackupScheduleChaining.func1
| pkg/cmd/roachtest/tests/backup_chaining.go:227
| main.(*testRunner).runTest.func2
| pkg/cmd/roachtest/test_runner.go:1558
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) backup jobs failed: [{1210965887987908610 failed backup exhausted retries in initial state: running distributed backup to export 27 ranges: exporting /Tenant/3/Table/114/1/{2/50001-4/51707}: batch timestamp 1789627730.697592000,0 must be after replica GC threshold 1789627744.481029835,0 (r127: /Tenant/3/Table/114/1/{2/50001-4/51707})}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: backup/schedule-chaining/tpcc-10
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/backup/schedule-chaining/tpcc-10/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21520622-1789624758-21-n5cpu8-0001 | | 10.91.100.155 |
| teamcity-21520622-1789624758-21-n5cpu8-0002 | | 10.91.96.191 |
| teamcity-21520622-1789624758-21-n5cpu8-0003 | | 10.91.98.53 |
| teamcity-21520622-1789624758-21-n5cpu8-0004 | | 10.91.100.154 |
| teamcity-21520622-1789624758-21-n5cpu8-0005 | | 10.91.100.205 |

Parameters:
- arch=arm64
- cloud=gce
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=true
- fs=ext4
- localSSD=false
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21520622/backup-schedule-chaining-tpcc-10/1789627661907/1789628386014)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21520622-1789624758-21-n5cpu8&from_ts=1789627661907&to_ts=1789628386014&live=false&storage=flex_tier)

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*backup/schedule-chaining/tpcc-10.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68467

Contributor guide

Open the contributing guide

Research direction

Start with the failure artifacts and logs for backup/schedule-chaining/tpcc-10, then read backup_chaining.go:227 and backup_fixtures.go:394 around registerBackupScheduleChaining and monitorBackups. Reproduce or inspect the arm64 GCE run and determine why the backup encounters the replica GC threshold error. Done means the test completes without the reported backup failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.