cockroachdb / cockroachdb/cockroach

release-26.2.5-rc: kv/kvserver/loqrecovery: TestRetrieveApplyStatus failed

Open
#172,562 4 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2.5-rc C-test-failure O-robot T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

kv/kvserver/loqrecovery.TestRetrieveApplyStatus [failed](https://mesolite.cluster.engflow.com/invocations/default/9590c3c2-78cf-43f9-bbaa-3184ff494d4b?testReportRun=2&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXIvbG9xcmVjb3Zlcnk6bG9xcmVjb3ZlcnlfdGVzdA==) on release-26.2.5-rc @ [aecc5c2a65078594fed36cb70bdbf2b0a0aaaf96](https://github.com/cockroachdb/cockroach/commits/aecc5c2a65078594fed36cb70bdbf2b0a0aaaf96):

```
internal/poll.(*pollDesc).wait(0xc017b82c80?, 0xc0056e0800?, 0x0)
GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc017b82c80, {0xc0056e0800, 0x800, 0x800})
GOROOT/src/internal/poll/fd_unix.go:165 +0x279
net.(*netFD).Read(0xc017b82c80, {0xc0056e0800?, 0x7fb?, 0xc01a7eb700?})
GOROOT/src/net/fd_posix.go:68 +0x25
net.(*conn).Read(0xc01218d6c0, {0xc0056e0800?, 0xc0056e0800?, 0x5?})
GOROOT/src/net/net.go:196 +0x45
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc02d4b9b00, {0xc0056e0800, 0x800, 0x800})
external/com_github_cockroachdb_cmux/cmux.go:241 +0x14f
crypto/tls.(*atLeastReader).Read(0xc018115170, {0xc0056e0800?, 0x7fb?, 0x2?})
GOROOT/src/crypto/tls/conn.go:816 +0x3b
bytes.(*Buffer).ReadFrom(0xc025884628, {0x6f4c840, 0xc018115170})
GOROOT/src/bytes/buffer.go:217 +0x98
crypto/tls.(*Conn).readFromUntil(0xc025884388, {0x7f12e9ea8230, 0xc02d4b9b00}, 0x47f934?)
GOROOT/src/crypto/tls/conn.go:838 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc025884388, 0x0)
GOROOT/src/crypto/tls/conn.go:627 +0x3db
crypto/tls.(*Conn).readRecord(...)
GOROOT/src/crypto/tls/conn.go:589
crypto/tls.(*Conn).Read(0xc025884388, {0xc0042de000, 0x1000, 0x1d5c9af?})
GOROOT/src/crypto/tls/conn.go:1392 +0x145
storj.io/drpc/drpcwire.(*Reader).read(0xc034800d90, {0xc0042de000, 0x1000, 0x1000})
external/io_storj_drpc/drpcwire/reader.go:71 +0x71
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0xc034800d90, {0xc01a444620?, 0x0?, 0xe0?})
external/io_storj_drpc/drpcwire/reader.go:116 +0x4ec
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc024ce0780)
external/io_storj_drpc/drpcmanager/manager.go:236 +0xd9
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 2143728
external/io_storj_drpc/drpcmanager/manager.go:128 +0x40c

goroutine 1935209 [select, 4 minutes]:
storj.io/drpc/drpcmanager.(*Manager).NewServerStream(0xc01e9ecf00, {0x6f92758, 0xc03a9800f0})
external/io_storj_drpc/drpcmanager/manager.go:461 +0x26e
storj.io/drpc/drpcserver.(*Server).ServeOne(0xc02580a340, {0x6f95078?, 0xc027430ab0?}, {0x6f84c80?, 0xc037a0ea88?})
external/io_storj_drpc/drpcserver/server.go:173 +0x4e7
storj.io/drpc/drpcserver.(*Server).Serve.func2({0x6f95078?, 0xc027430ab0?})
external/io_storj_drpc/drpcserver/server.go:229 +0x57
storj.io/drpc/drpcctx.(*Tracker).track(0xc027430ab0, 0x0?)
external/io_storj_drpc/drpcctx/tracker.go:35 +0x25
created by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 218834
external/io_storj_drpc/drpcctx/tracker.go:30 +0x76

goroutine 2034711 [select, 4 minutes]:
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc01d6f7a40)
external/io_storj_drpc/drpcmanager/manager.go:327 +0x110
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 201786
external/io_storj_drpc/drpcmanager/manager.go:129 +0x448
```

Parameters:
- attempt=1
- run=2
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #171452 release-26.2.2-rc: kv/kvserver/loqrecovery: TestRetrieveApplyStatus failed [A-testing C-bug C-test-failure O-robot P-3 T-kv branch-release-26.2.2-rc]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestRetrieveApplyStatus.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65810

Contributor guide

Open the contributing guide

Research direction

Start with kv/kvserver/loqrecovery.TestRetrieveApplyStatus and inspect the linked test invocation and stack trace. Review the internal Go test-failure investigation guide and compare the related failure in issue #171452; done means the failure’s cause is identified and the test completes successfully on the affected release branch.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.