cockroachdb / cockroachdb/cockroach

basalt-26.3.2-rc-20260908: pkg/ccl/serverccl/statusccl/statusccl_test: TestTenantStatusAPI failed

Open
#174,969 0 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3.2-rc-20260908 C-test-failure O-robot release-blocker T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI [failed](https://mesolite.cluster.engflow.com/invocations/default/832f0de3-0bfe-4869-aceb-5624ee5c5bd4?testReportRun=2&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvY2NsL3NlcnZlcmNjbC9zdGF0dXNjY2w6c3RhdHVzY2NsX3Rlc3Q=) on basalt-26.3.2-rc-20260908 @ [807558e5e81666af257c811e925e4cae570abc28](https://github.com/cockroachdb/cockroach/commits/807558e5e81666af257c811e925e4cae570abc28):

Fatal error:

```
panic: pebble: blob reference value size 13713 is greater than the blob file's value size 13685

```
Stack:

```
goroutine 607 [running]:
github.com/cockroachdb/pebble/internal/manifest.MakeBlobReference(0x118940547d78?, 0x11894bdff7a0?, 0x3?, 0x11894c24f6d0)
external/com_github_cockroachdb_pebble/internal/manifest/blob_metadata.go:60 +0x17f
github.com/cockroachdb/pebble/internal/manifest.(*BulkVersionEdit).Apply(0x118947705e00, 0x11893fa0b008, 0x3e80)
external/com_github_cockroachdb_pebble/internal/manifest/version_edit.go:1512 +0xd2f
github.com/cockroachdb/pebble.(*rsManifestInstaller).prepareExternalManifest(0x0?, 0xfce, 0x11893fa0b008)
external/com_github_cockroachdb_pebble/manifest_installer.go:830 +0x1ab1
github.com/cockroachdb/pebble.(*rsManifestInstaller).PrepareExternalManifest.func1(...)
external/com_github_cockroachdb_pebble/manifest_installer.go:400
github.com/cockroachdb/pebble.(*rsManifestInstaller).PrepareExternalManifest(0x11894c06d968, 0xfce)
external/com_github_cockroachdb_pebble/manifest_installer.go:409 +0x194
github.com/cockroachdb/pebble.(*DB).PrepareExternalManifest(...)
external/com_github_cockroachdb_pebble/db.go:3593
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleRSEngine).prepareExternalManifest(0x0?, 0x118947706708?)
pkg/storage/rs_engine_pebble.go:146 +0x30
github.com/cockroachdb/cockroach/pkg/storage.(*rsEngineContainer).PrepareExternalManifest(0x118948c84c88, 0xfce, 0x1)
pkg/storage/rs_engine_container.go:531 +0x113
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine(0x11894c0301a0, {0x804b100, 0x11894c3aed20})
pkg/kv/kvserver/replica_app_batch.go:667 +0x1b4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0x118947707470, {0x804b100, 0x11894c3aed20}, {0x80898c0, 0x11894c0305d8})
pkg/kv/kvserver/apply/task.go:285 +0x153
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0x118947707470, {0x804b100, 0x11894c3aed20})
pkg/kv/kvserver/apply/task.go:246 +0x92
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:1246 +0x1c25
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:881 +0x1d0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x11892dbc2008, 0x11892cde2ab0?)
pkg/kv/kvserver/store_raft.go:766 +0x15b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x11892da56460, {0x804b100, 0x11892c6b3540}, {0x8077c40, 0x11892dbc2008}, 0x11892dbb9108)
pkg/kv/kvserver/scheduler.go:421 +0x2b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x804b100, 0x11892c6b3540}, 0x11892d50d8b0?)
pkg/kv/kvserver/scheduler.go:314 +0xe9
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 483
pkg/kv/kvserver/scheduler.go:329 +0x27e
```
Log preceding fatal error

```
=== RUN TestTenantStatusAPI
test_log_scope.go:172: test logs captured to: outputs.zip/logTestTenantStatusAPI3189920514
=== RUN TestTenantStatusAPI/reset_sql_stats
=== RUN TestTenantStatusAPI/reset_sql_stats/flushed=false
```

Parameters:
- attempt=1
- run=2
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/obs-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestTenantStatusAPI.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing pkg/ccl/serverccl/statusccl/statusccl_test.TestTenantStatusAPI, using the linked test report and the captured logs. Trace the panic from pkg/storage/rs_engine_pebble.go and pkg/kv/kvserver/replica_app_batch.go into Pebble's manifest code. Done means the test completes without the blob-reference size panic and includes coverage for the failure if a regression test is practical.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.