cockroachdb / cockroachdb/cockroach

roachtest: schemachange/bulkingest/nodes=12/rows=4000000000/create_index failed

Open
#170,457 8 comments 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.4 C-test-failure O-roachtest O-robot T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.schemachange/bulkingest/nodes=12/rows=4000000000/create_index [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21350483?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21350483?buildTab=artifacts#/schemachange/bulkingest/nodes=12/rows=4000000000/create_index) on release-25.4 @ [12d38425f6027fbb20aeddd56424e7805e89bb5a](https://github.com/cockroachdb/cockroach/commits/12d38425f6027fbb20aeddd56424e7805e89bb5a):

Failed with:

```
(schemachange.go:433).2: dial tcp 34.148.60.36:26257: connect: connection refused
(monitor.go:267).Wait: monitor failure: full command output in run_140704.611999991_n12_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(metric_utils.go:206).CloseExporter: failed to export perf stats: context canceled
(test_runner.go:1881).maybeSaveClusterDueToInvariantProblems: invariant problem - snap name :
logs/cockroach.log:F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ?crc32c? checksum mismatch 0 != bc85bb33
logs/cockroach.teamcity-21350483-1778997858-197-n12cpu4-0001.ubuntu.2026-05-17T13_52_09Z.005417.log:F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ?crc32c? checksum mismatch 0 != bc85bb33
test artifacts and logs in: /artifacts/schemachange/bulkingest/nodes=12/rows=4000000000/create_index/run_1
```
Fatal entries found in Cockroach logs:

```
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ‹crc32c› checksum mismatch 0 != bc85bb33
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +(1) ‹path: /mnt/data1/cockroach/003734.sst›
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +Wraps: (2) attached stack trace
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + -- stack trace:
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:524
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:450
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/reader.go:394
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 local corruption detected: pebble: file 003734: block 2055918/25849: ‹crc32c› checksum mismatch 0 != bc85bb33
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +(1) ‹path: /mnt/data1/cockroach/003734.sst›
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 +Wraps: (2) attached stack trace
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + -- stack trace:
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:524
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/block/block.go:450
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
F260517 14:10:55.601465 107676 storage/pebble.go:1313 ⋮ [n1,s1,pebble] 510 + | external/com_github_cockroachdb_pebble/sstable/reader.go:394
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- saved=invariant problem - snap name <failed>
- ssd=4
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21350483/schemachange-bulkingest-nodes-12-rows-4000000000-create-index/1779025915612/1779027290705)

/cc @cockroachdb/sql-foundations

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*schemachange/bulkingest/nodes=12/rows=4000000000/create_index.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-64018

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.