cockroachdb / cockroachdb/cockroach

workload: import fails during validation with: batch timestamp must be after replica GC threshold

Open
#110,738 9 comments 0 reactions 0 assignees View on GitHub
C-bug O-community T-sql-queries X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

I try to import initial TPC-C data.

```
cockroach workload fixtures import tpcc --warehouses=13000 'postgres://root@host:26257?sslmode=disable'

I230915 14:37:47.148627 1 ccl/workloadccl/fixture.go:315 [-] 1 starting import of 9 tables
I230915 14:37:49.950348 108 ccl/workloadccl/fixture.go:492 [-] 2 imported 705 KiB in warehouse table (13000 rows, 0 index entries, took 2.622700443s, 0.26 MiB/s)
I230915 14:37:51.202532 109 ccl/workloadccl/fixture.go:492 [-] 3 imported 13 MiB in district table (130000 rows, 0 index entries, took 3.874756765s, 3.34 MiB/s)
I230915 14:37:52.557898 226 ccl/workloadccl/fixture.go:492 [-] 4 imported 7.9 MiB in item table (100000 rows, 0 index entries, took 5.229539536s, 1.50 MiB/s)
I230915 14:38:32.296802 113 ccl/workloadccl/fixture.go:492 [-] 5 imported 1.7 GiB in new_order table (117000000 rows, 0 index entries, took 44.968524969s, 39.63 MiB/s)
I230915 14:43:16.869316 112 ccl/workloadccl/fixture.go:492 [-] 6 imported 21 GiB in order table (390000000 rows, 390000000 index entries, took 5m29.541104727s, 65.95 MiB/s)
I230915 14:44:10.181317 111 ccl/workloadccl/fixture.go:492 [-] 7 imported 28 GiB in history table (390000000 rows, 0 index entries, took 6m22.853216195s, 76.12 MiB/s)
I230915 14:49:05.337496 110 ccl/workloadccl/fixture.go:492 [-] 8 imported 225 GiB in customer table (390000000 rows, 390000000 index entries, took 11m18.009532307s, 340.00 MiB/s)
I230915 14:53:58.334377 227 ccl/workloadccl/fixture.go:492 [-] 9 imported 392 GiB in stock table (1300000000 rows, 0 index entries, took 16m11.005846138s, 413.32 MiB/s)
I230915 14:56:16.755741 228 ccl/workloadccl/fixture.go:492 [-] 10 imported 225 GiB in order_line table (3899967475 rows, 0 index entries, took 18m29.427167889s, 207.68 MiB/s)
I230915 14:56:16.812441 1 ccl/workloadccl/fixture.go:323 [-] 11 imported 894 GiB bytes in 9 tables (took 18m29.663295738s, 824.53 MiB/s)
I230915 14:56:19.290831 1 ccl/workloadccl/cliccl/fixtures.go:343 [-] 12 fixture is restored; now running consistency checks (ctrl-c to abort)
I230915 14:56:19.504132 1 workload/tpcc/tpcc.go:517 [-] 13 check 3.3.2.1 took 213.183355ms
I230915 14:58:37.656937 1 workload/tpcc/tpcc.go:517 [-] 14 check 3.3.2.2 took 2m18.152599629s
I230915 14:59:00.109822 1 workload/tpcc/tpcc.go:517 [-] 15 check 3.3.2.3 took 22.452724373s

I230915 15:11:48.106174 1 workload/tpcc/tpcc.go:517 [-] 16 check 3.3.2.4 took 12m47.996230562s
Error: check failed: 3.3.2.4: at 1694789940110274588.0000000000: length of order.sum(o_ol_cnt) != order_line.count(*)
```

Later I tried to do a quick check on my own:

```
root@localhost:26257/defaultdb> select sum(o_ol_cnt) from tpcc.order;
sum
--------------
3899967475
(1 row)

Time: 97.993s total (execution 97.992s / network 0.001s)

root@localhost:26257/defaultdb> select count(*) from tpcc.order_line ;
count
--------------
3899967475
(1 row)

Time: 320.052s total (execution 320.052s / network 0.001s)

```

The check is not per warehous/district though, but total number of order lines is correct.

**Environment:**
- v23-1-9
- Ubuntu 20.04

**Additional context**

My colleague also faced with import issues:
`Error: check failed: 3.3.2.7: pq: batch timestamp 1690649337.212712683,0 must be after replica GC threshold 1690649369.613864206,0 `

Jira issue: CRDB-31582

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.