cockroachdb / cockroachdb/cockroach
basalt-26.3.2-rc-20260908: pkg/ccl/serverccl/adminccl/adminccl_test: TestTenantAdminAPI failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
pkg/ccl/serverccl/adminccl/adminccl_test.TestTenantAdminAPI [failed](https://mesolite.cluster.engflow.com/invocations/default/832f0de3-0bfe-4869-aceb-5624ee5c5bd4?testReportRun=3&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvY2NsL3NlcnZlcmNjbC9hZG1pbmNjbDphZG1pbmNjbF90ZXN0) on basalt-26.3.2-rc-20260908 @ [807558e5e81666af257c811e925e4cae570abc28](https://github.com/cockroachdb/cockroach/commits/807558e5e81666af257c811e925e4cae570abc28):
Fatal error:
```
panic: pebble: blob reference value size 16279 is greater than the blob file's value size 13686
```
Stack:
```
goroutine 598 [running]:
github.com/cockroachdb/pebble/internal/manifest.MakeBlobReference(0x1ae5083c1578?, 0x1ae5084e4fc0?, 0x3?, 0x1ae508118640)
external/com_github_cockroachdb_pebble/internal/manifest/blob_metadata.go:60 +0x17f
github.com/cockroachdb/pebble/internal/manifest.(*BulkVersionEdit).Apply(0x1ae4f8b5de00, 0x1ae508321c08, 0x3e80)
external/com_github_cockroachdb_pebble/internal/manifest/version_edit.go:1512 +0xd2f
github.com/cockroachdb/pebble.(*rsManifestInstaller).prepareExternalManifest(0x0?, 0xa24, 0x1ae508321c08)
external/com_github_cockroachdb_pebble/manifest_installer.go:830 +0x1ab1
github.com/cockroachdb/pebble.(*rsManifestInstaller).PrepareExternalManifest.func1(...)
external/com_github_cockroachdb_pebble/manifest_installer.go:400
github.com/cockroachdb/pebble.(*rsManifestInstaller).PrepareExternalManifest(0x1ae508357668, 0xa24)
external/com_github_cockroachdb_pebble/manifest_installer.go:409 +0x194
github.com/cockroachdb/pebble.(*DB).PrepareExternalManifest(...)
external/com_github_cockroachdb_pebble/db.go:3593
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleRSEngine).prepareExternalManifest(0x0?, 0x1ae4f8b5e708?)
pkg/storage/rs_engine_pebble.go:146 +0x30
github.com/cockroachdb/cockroach/pkg/storage.(*rsEngineContainer).PrepareExternalManifest(0x1ae508424008, 0xa24, 0x1)
pkg/storage/rs_engine_container.go:531 +0x113
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine(0x1ae5083551a0, {0x802b5f0, 0x1ae5083857c0})
pkg/kv/kvserver/replica_app_batch.go:667 +0x1b4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0x1ae4f8b5f470, {0x802b5f0, 0x1ae5083857c0}, {0x806a918, 0x1ae5083555d8})
pkg/kv/kvserver/apply/task.go:285 +0x153
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0x1ae4f8b5f470, {0x802b5f0, 0x1ae5083857c0})
pkg/kv/kvserver/apply/task.go:246 +0x92
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:1246 +0x1c25
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:881 +0x1d0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x1ae4e7f58008, 0x1ae4e72da3c0?)
pkg/kv/kvserver/store_raft.go:766 +0x15b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x1ae4e8bb9b90, {0x802b5f0, 0x1ae4e70a4f00}, {0x8059e00, 0x1ae4e7f58008}, 0x1ae4e7f69108)
pkg/kv/kvserver/scheduler.go:421 +0x2b6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x802b5f0, 0x1ae4e70a4f00}, 0x1ae4e8367720?)
pkg/kv/kvserver/scheduler.go:314 +0xe9
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 476
pkg/kv/kvserver/scheduler.go:329 +0x27e
```
Log preceding fatal error
```
=== RUN TestTenantAdminAPI
test_log_scope.go:172: test logs captured to: outputs.zip/logTestTenantAdminAPI1012181491
test_log_scope.go:82: use -show-logs to present logs inline
=== RUN TestTenantAdminAPI/tenant_jobs
```
Parameters:
- attempt=1
- run=3
- shard=1
Help
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
/cc @cockroachlabs/unowned
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestTenantAdminAPI.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Contributor guide
Research direction
Start by running pkg/ccl/serverccl/adminccl/adminccl_test.TestTenantAdminAPI and inspect the linked test report and captured logs. Trace the panic through pkg/storage/rs_engine_pebble.go, pkg/storage/rs_engine_container.go, and pkg/kv/kvserver/replica_app_batch.go into Pebble's manifest installation. Done means the tenant_jobs test completes without the blob reference size panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100