cockroachdb / cockroachdb/cockroach
basalt-26.3.2-rc-20260914: kv/kvserver: TestAckWriteBeforeApplication failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
kv/kvserver.TestAckWriteBeforeApplication [failed](https://mesolite.cluster.engflow.com/invocations/default/36252f8d-8f49-40a2-b8b8-d6dd84ae76b7?testReportRun=1&testReportShard=14&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on basalt-26.3.2-rc-20260914 @ [d3eb08ebdd1dacc8d3dac0b107a7db5929e09969](https://github.com/cockroachdb/cockroach/commits/d3eb08ebdd1dacc8d3dac0b107a7db5929e09969):
```
goroutine 24580 [select, 15 minutes, "job":"AUTO ASH COMPACTION id=108", "n":"1"]:
github.com/cockroachdb/cockroach/pkg/sql.(*ashCompactionResumer).Resume(0xc01fbdcd90, {0xd2dd0f0, 0xc0286d4600}, {0xc8da580, 0xc01defad10})
pkg/sql/compact_ash.go:56 +0x4d5
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc028e8de00, 0xc02674ea88, 0xc02d917708, {0xd2c2230, 0xc01fbdcd90}, {0xd2dd0f0, 0xc0286d4600}, {0xc8da580, 0xc01defad10})
pkg/jobs/registry.go:1691 +0x16b
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc02674ea88, {0xd2dd0f0, 0xc0286d42d0}, {0xc8da580, 0xc01defad10}, {0xd2c2230, 0xc01fbdcd90}, 0xc019f52d80, {0xc028bed879, 0x7}, ...)
pkg/jobs/registry.go:1692 +0x1268
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc02674ea88, {0xd2dd0f0, 0xc0286d42d0}, {0xd2c2230, 0xc01fbdcd90}, 0xc019f52d80, {0xc028bed879, 0x7}, {0xc01b348c89, 0x7})
pkg/jobs/adopt.go:503 +0x82b
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0xd2dd0f0, 0xc02be25800})
pkg/jobs/adopt.go:350 +0x199
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd2dd0f0, 0xc02be25800})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 24529
pkg/util/stop/stopper.go:445 +0x247
goroutine 23388 [select, 75 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*testServer).Activate.func2()
pkg/server/testserver.go:999 +0x1a9
created by github.com/cockroachdb/cockroach/pkg/server.(*testServer).Activate in goroutine 20581
pkg/server/testserver.go:996 +0x2d5
goroutine 21351 [select]:
github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
pkg/util/admission/work_queue.go:646 +0x1c9
created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 20581
pkg/util/admission/work_queue.go:617 +0x8b
goroutine 23205 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher[...]).Run(0xd328700, {0xd2dd0b8, 0xc01b77fdb0})
pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:344 +0x10b9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start[...].func1()
pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:202 +0x413
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd2dd898, 0xc0032dc500})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 20581
pkg/util/stop/stopper.go:445 +0x247
goroutine 21979 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*bufferedRegistration).outputLoop(0xc023c59c00, {0xd2dd0b8, 0xc01a3c9220})
pkg/kv/kvserver/rangefeed/buffered_registration.go:246 +0x35d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*bufferedRegistration).runOutputLoop(0xc023c59c00, {0xd2dd898, 0xc01a397540}, 0xc01c012ce8?)
pkg/kv/kvserver/rangefeed/buffered_registration.go:280 +0x191
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1({0xd2dd898, 0xc01a397540})
pkg/kv/kvserver/rangefeed/scheduled_processor.go:377 +0x73
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd2dd898, 0xc01a397540})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 21090
pkg/util/stop/stopper.go:445 +0x247
```
Parameters:
- attempt=1
- race=true
- run=1
- shard=14
Help
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestAckWriteBeforeApplication.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Contributor guide
Research direction
Start with kv/kvserver.TestAckWriteBeforeApplication and the linked test report, using the race=true parameters to reproduce the failure. Inspect the reported goroutine stack and the referenced kvserver paths to determine why the test did not complete; done means the underlying failure is addressed and the test passes under the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100