cockroachdb / cockroachdb/cockroach
roachtest: schemachange/bulkingest/nodes=12/rows=4000000000/create_index failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.schemachange/bulkingest/nodes=12/rows=4000000000/create_index [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21350483?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21350483?buildTab=artifacts#/schemachange/bulkingest/nodes=12/rows=4000000000/create_index) on release-25.4 @ [12d38425f6027fbb20aeddd56424e7805e89bb5a](https://github.com/cockroachdb/cockroach/commits/12d38425f6027fbb20aeddd56424e7805e89bb5a):
Failed with:
```
(schemachange.go:433).2: dial tcp 34.148.60.36:26257: connect: connection refused
(monitor.go:267).Wait: monitor failure: full command output in run_140704.611999991_n12_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(metric_utils.go:206).CloseExporter: failed to export perf stats: context canceled
(test_runner.go:1881).maybeSaveClusterDueToInvariantProblems: invariant problem - snap name :
logs/cockroach.log:F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ?crc32c? checksum mismatch 0 != bc85bb33
logs/cockroach.teamcity-21350483-1778997858-197-n12cpu4-0001.ubuntu.2026-05-17T13_52_09Z.005417.log:F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ?crc32c? checksum mismatch 0 != bc85bb33
test artifacts and logs in: /artifacts/schemachange/bulkingest/nodes=12/rows=4000000000/create_index/run_1
```
Fatal entries found in Cockroach logs:
```
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ‹crc32c› checksum mismatch 0 != bc85bb33
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +(1) ‹path: /mnt/data1/cockroach/003734.sst›
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +Wraps: (2) attached stack trace
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + -- stack trace:
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:524
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:450
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/reader.go:394
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ‹crc32c› checksum mismatch 0 != bc85bb33
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +(1) ‹path: /mnt/data1/cockroach/003734.sst›
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +Wraps: (2) attached stack trace
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + -- stack trace:
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:524
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:450
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/reader.go:394
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- saved=invariant problem - snap name <failed>
- ssd=4
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21350483/schemachange-bulkingest-nodes-12-rows-4000000000-create-index/1779025915612/1779027290705)
/cc @cockroachdb/sql-foundations
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*schemachange/bulkingest/nodes=12/rows=4000000000/create_index.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-64018
Contributor guide
Assessment
This issue has not been assessed yet.