cockroachdb / cockroachdb/cockroach

basalt-26.3.2-rc-20260914: kv/kvserver: TestAckWriteBeforeApplication failed

Open
#175,625 1 comment 0 reactions 1 assignee Assigned to @iskettaneh View on GitHub
A-testing branch-basalt-26.3.2-rc-20260914 C-bug C-test-failure O-robot T-kv X-duplicate
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

kv/kvserver.TestAckWriteBeforeApplication [failed](https://mesolite.cluster.engflow.com/invocations/default/36252f8d-8f49-40a2-b8b8-d6dd84ae76b7?testReportRun=1&testReportShard=14&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on basalt-26.3.2-rc-20260914 @ [d3eb08ebdd1dacc8d3dac0b107a7db5929e09969](https://github.com/cockroachdb/cockroach/commits/d3eb08ebdd1dacc8d3dac0b107a7db5929e09969):

```

goroutine 24580 [select, 15 minutes, "job":"AUTO ASH COMPACTION id=108", "n":"1"]:
github.com/cockroachdb/cockroach/pkg/sql.(*ashCompactionResumer).Resume(0xc01fbdcd90, {0xd2dd0f0, 0xc0286d4600}, {0xc8da580, 0xc01defad10})
pkg/sql/compact_ash.go:56 +0x4d5
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc028e8de00, 0xc02674ea88, 0xc02d917708, {0xd2c2230, 0xc01fbdcd90}, {0xd2dd0f0, 0xc0286d4600}, {0xc8da580, 0xc01defad10})
pkg/jobs/registry.go:1691 +0x16b
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc02674ea88, {0xd2dd0f0, 0xc0286d42d0}, {0xc8da580, 0xc01defad10}, {0xd2c2230, 0xc01fbdcd90}, 0xc019f52d80, {0xc028bed879, 0x7}, ...)
pkg/jobs/registry.go:1692 +0x1268
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc02674ea88, {0xd2dd0f0, 0xc0286d42d0}, {0xd2c2230, 0xc01fbdcd90}, 0xc019f52d80, {0xc028bed879, 0x7}, {0xc01b348c89, 0x7})
pkg/jobs/adopt.go:503 +0x82b
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0xd2dd0f0, 0xc02be25800})
pkg/jobs/adopt.go:350 +0x199
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd2dd0f0, 0xc02be25800})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 24529
pkg/util/stop/stopper.go:445 +0x247

goroutine 23388 [select, 75 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*testServer).Activate.func2()
pkg/server/testserver.go:999 +0x1a9
created by github.com/cockroachdb/cockroach/pkg/server.(*testServer).Activate in goroutine 20581
pkg/server/testserver.go:996 +0x2d5

goroutine 21351 [select]:
github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
pkg/util/admission/work_queue.go:646 +0x1c9
created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 20581
pkg/util/admission/work_queue.go:617 +0x8b

goroutine 23205 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher[...]).Run(0xd328700, {0xd2dd0b8, 0xc01b77fdb0})
pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:344 +0x10b9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start[...].func1()
pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:202 +0x413
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd2dd898, 0xc0032dc500})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 20581
pkg/util/stop/stopper.go:445 +0x247

goroutine 21979 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*bufferedRegistration).outputLoop(0xc023c59c00, {0xd2dd0b8, 0xc01a3c9220})
pkg/kv/kvserver/rangefeed/buffered_registration.go:246 +0x35d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*bufferedRegistration).runOutputLoop(0xc023c59c00, {0xd2dd898, 0xc01a397540}, 0xc01c012ce8?)
pkg/kv/kvserver/rangefeed/buffered_registration.go:280 +0x191
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1({0xd2dd898, 0xc01a397540})
pkg/kv/kvserver/rangefeed/scheduled_processor.go:377 +0x73
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd2dd898, 0xc01a397540})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 21090
pkg/util/stop/stopper.go:445 +0x247
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=14
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestAckWriteBeforeApplication.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with kv/kvserver.TestAckWriteBeforeApplication and the linked test report, using the race=true parameters to reproduce the failure. Inspect the reported goroutine stack and the referenced kvserver paths to determine why the test did not complete; done means the underlying failure is addressed and the test passes under the reported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.