cockroachdb / cockroachdb/cockroach
tpcc server-side partitioning (`partitionTables`) is slow
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
During the initial load, we observed that post-processing, majority of it is spent in `partitionTables`, took nearly `11 hours` 😱 , while the load itself was completed in under 3 hours,
```
Dec 08 05:42:43 workload-scale-300-0001 bash[4106]: I251208 05:42:43.966335 1 ccl/workloadccl/fixture.go:321 [-] 1 starting import of 9 tables
Dec 08 05:42:50 workload-scale-300-0001 bash[4106]: I251208 05:42:50.873454 71 ccl/workloadccl/fixture.go:521 [-] 2 imported 8.7 MiB in item table (100000 rows, 0 index entries, took 6.051637369s, 1.44 MiB/s)
Dec 08 05:45:05 workload-scale-300-0001 bash[4106]: I251208 05:45:05.134738 33 ccl/workloadccl/fixture.go:521 [-] 3 imported 246 MiB in warehouse table (4000000 rows, 0 index entries, took 2m20.317415898s, 1.76 MiB/s)
Dec 08 05:47:23 workload-scale-300-0001 bash[4106]: I251208 05:47:23.876273 66 ccl/workloadccl/fixture.go:521 [-] 4 imported 4.2 GiB in district table (40000000 rows, 0 index entries, took 4m39.058122092s, 15.50 MiB/s)
Dec 08 05:49:02 workload-scale-300-0001 bash[4106]: I251208 05:49:02.023229 70 ccl/workloadccl/fixture.go:521 [-] 5 imported 871 GiB in new_order table (36000000000 rows, 0 index entries, took 6m17.202159524s, 2364.97 MiB/s)
Dec 08 06:39:42 workload-scale-300-0001 bash[4106]: I251208 06:39:42.106210 68 ccl/workloadccl/fixture.go:521 [-] 6 imported 9.6 TiB in history table (120000000000 rows, 0 index entries, took 56m57.286552286s, 2938.03 MiB/s)
Dec 08 06:42:20 workload-scale-300-0001 bash[4106]: I251208 06:42:20.288744 69 ccl/workloadccl/fixture.go:521 [-] 7 imported 8.6 TiB in order table (120000000000 rows, 120000000000 index entries, took 59m35.4683925s, 2511.14 MiB/s)
Dec 08 07:32:19 workload-scale-300-0001 bash[4106]: I251208 07:32:19.518821 67 ccl/workloadccl/fixture.go:521 [-] 8 imported 70 TiB in customer table (120000000000 rows, 120000000000 index entries, took 1h49m34.699905232s, 11102.17 MiB/s)
Dec 08 08:08:59 workload-scale-300-0001 bash[4106]: I251208 08:08:59.625228 72 ccl/workloadccl/fixture.go:521 [-] 9 imported 121 TiB in stock table (400000000000 rows, 0 index entries, took 2h26m14.802701664s, 14508.40 MiB/s)
Dec 08 08:31:16 workload-scale-300-0001 bash[4106]: I251208 08:31:16.260931 73 ccl/workloadccl/fixture.go:521 [-] 10 imported 80 TiB in order_line table (1199998407174 rows, 0 index entries, took 2h48m31.437730453s, 8299.72 MiB/s)
Dec 08 08:31:16 workload-scale-300-0001 bash[4106]: I251208 08:31:16.284273 1 ccl/workloadccl/fixture.go:329 [-] 11 imported 290 TiB bytes in 9 tables (took 2h48m32.317762833s, 30076.10 MiB/s)
Dec 08 18:22:07 workload-scale-300-0001 systemd[1]: tpcc_init_cct_tpcc.service: Deactivated successfully
```
Jira issue: CRDB-58226
Contributor guide
Assessment
This issue has not been assessed yet.