cockroachdb / cockroachdb/cockroach

release-26.3.0-rc: kv/kvserver: TestLogGrowthWhenRefreshingPendingCommands failed

Open
#172,958 3 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3.0-rc C-test-failure O-robot P-3 T-kv X-duplicate
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

kv/kvserver.TestLogGrowthWhenRefreshingPendingCommands [failed](https://mesolite.cluster.engflow.com/invocations/default/ed1955e8-5e64-46d0-a0fa-77bd2c0492f7?testReportRun=1&testReportShard=23&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on release-26.3.0-rc @ [0a3dc2524e335b85175c507a318dd57f28f92ef3](https://github.com/cockroachdb/cockroach/commits/0a3dc2524e335b85175c507a318dd57f28f92ef3):

```
=== RUN TestLogGrowthWhenRefreshingPendingCommands
test_log_scope.go:172: test logs captured to: outputs.zip/logTestLogGrowthWhenRefreshingPendingCommands3283699049
test_log_scope.go:82: use -show-logs to present logs inline
client_raft_test.go:2055: -- test log scope end --
test logs left over in: outputs.zip/logTestLogGrowthWhenRefreshingPendingCommands3283699049
--- FAIL: TestLogGrowthWhenRefreshingPendingCommands (162.58s)
=== RUN TestLogGrowthWhenRefreshingPendingCommands/lease-type=LeaseExpiration
--- FAIL: TestLogGrowthWhenRefreshingPendingCommands/lease-type=LeaseExpiration (58.15s)
=== RUN TestLogGrowthWhenRefreshingPendingCommands/lease-type=LeaseExpiration/proposeOnFollower=true
test_server_shim.go:146: using test tenant configuration from testserver factory defaults
test_server_shim.go:639: DRPC is enabled (via testserver factory defaults)
client_raft_test.go:2043: write finished with quorum unavailable; err=[NotLeaseHolderError] refusing to acquire lease on follower; r88: replica (n2,s2):2 not lease holder; current lease is repl=(n1,s1):1 seq=0 start=0,0 type=LeaseExpiration exp= pro=0,0 acq=Unspecified
--- FAIL: TestLogGrowthWhenRefreshingPendingCommands/lease-type=LeaseExpiration/proposeOnFollower=true (29.33s)
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=23
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #170482 release-26.2: kv/kvserver: TestLogGrowthWhenRefreshingPendingCommands failed [A-testing C-bug C-test-failure O-robot P-3 T-kv X-nostale branch-release-26.2]
- #158589 kv/kvserver: TestLogGrowthWhenRefreshingPendingCommands failed [C-test-failure O-robot P-3 T-db-server branch-release-25.4.2-rc]
- #158047 kv/kvserver: TestLogGrowthWhenRefreshingPendingCommands failed [C-test-failure O-robot P-3 T-db-server branch-release-25.4.0-rc]
- #154843 kv/kvserver: TestLogGrowthWhenRefreshingPendingCommands failed [C-test-failure O-robot P-3 T-db-server branch-release-25.4]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestLogGrowthWhenRefreshingPendingCommands.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with TestLogGrowthWhenRefreshingPendingCommands in client_raft_test.go, especially the failure path around lines 2043-2055, and inspect the captured test logs from the linked invocation. Reproduce the race-enabled test on release-26.3.0-rc and compare the repeated failures on the listed release branches. Done means identifying why the follower write reports quorum or lease unavailability and making the test pass reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.