cockroachdb / cockroachdb/cockroach
roachtest: tpcc import workload should tolerate "node liveness error"
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The import may fail with "node liveness error: restarting in background" which may indicate a cluster overload. However, the job may complete on retry; further failure context should also be extracted from the failed job.
```
run_105938.201454743_n1_cockroach-workload-f: 10:59:38 cluster.go:2417: > ./cockroach workload fixtures import tpcc --checks=false --partitions=3 --zones="us-east1-b,us-west1-b,europe-west2-b" {pgurl:1} --warehouses 3000
I250628 10:59:39.057899 1 ccl/workloadccl/fixture.go:312 [-] 1 starting import of 9 tables
I250628 10:59:56.523490 59 ccl/workloadccl/fixture.go:489 [-] 2 imported 8.7 MiB in item table (100000 rows, 0 index entries, took 17.181702895s, 0.51 MiB/s)
I250628 10:59:59.688997 54 ccl/workloadccl/fixture.go:489 [-] 3 imported 3.2 MiB in district table (30000 rows, 0 index entries, took 20.347735599s, 0.16 MiB/s)
I250628 11:00:00.515136 53 ccl/workloadccl/fixture.go:489 [-] 4 imported 186 KiB in warehouse table (3000 rows, 0 index entries, took 21.173941402s, 0.01 MiB/s)
I250628 11:00:32.787650 58 ccl/workloadccl/fixture.go:489 [-] 5 imported 641 MiB in new_order table (27000000 rows, 0 index entries, took 53.445768368s, 11.99 MiB/s)
I250628 11:04:13.861913 57 ccl/workloadccl/fixture.go:489 [-] 6 imported 6.4 GiB in order table (90000000 rows, 90000000 index entries, took 4m34.520181636s, 23.84 MiB/s)
I250628 11:04:16.395950 56 ccl/workloadccl/fixture.go:489 [-] 7 imported 7.2 GiB in history table (90000000 rows, 0 index entries, took 4m37.054316531s, 26.65 MiB/s)
Error: importing fixture: importing table order_line: pq: job 1084745250814984193: node liveness error: restarting in background
run_105938.201454743_n1_cockroach-workload-f: 11:07:38 cluster.go:2427: > result: COMMAND_PROBLEM: exit status 1
```
[1] https://github.com/cockroachdb/cockroach/issues/149148
Jira issue: CRDB-52429
Contributor guide
Assessment
This issue has not been assessed yet.