cockroachdb / cockroachdb/cockroach

release-26.2: util/startup: TestStartupFailureRandomRange failed

Open
#172,780 3 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2 C-test-failure O-robot
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

util/startup.TestStartupFailureRandomRange [failed](https://mesolite.cluster.engflow.com/invocations/default/242a15b3-a450-4414-8072-ba10470dd1b0?testReportRun=3&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvdXRpbC9zdGFydHVwOnN0YXJ0dXBfdGVzdA==) on release-26.2 @ [914fcaa1a2da846f700b172e18381b6763054f37](https://github.com/cockroachdb/cockroach/commits/914fcaa1a2da846f700b172e18381b6763054f37):

```
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc003d5a000, {{0x1}, {0x6f90400, 0xc0012cd860}}, 0x0}, {0x6125bc0, 0xc004067c08}, {0x6f6aff0, 0xc02b79b4a0}, 0xc04e53b548, ...)
pkg/rpc/auth.go:192 +0x3a9
google.golang.org/grpc.getChainStreamHandler.func1({0x6125bc0?, 0xc004067c08?}, {0x6f6aff0?, 0xc02b79b4a0?})
external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc00326b170?, 0x6f33838?})
pkg/rpc/context.go:148 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00326b170, {0x6f33838, 0xc030ca7800}, {0xc03089c7b0, 0x27}, 0xc060b6aab0)
pkg/util/stop/stopper.go:358 +0x137
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x6125bc0, 0xc004067c08}, {0x6f6aff0, 0xc02b79b4a0}, 0xc04e53b548, 0xc020142bc0)
pkg/rpc/context.go:147 +0xcc
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x6125bc0, 0xc004067c08}, {0x6f6aff0, 0xc02b79b4a0}, 0xc04e53b548, 0xc01fbe0860?)
external/org_golang_google_grpc/server.go:1495 +0x7c
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0037921e0, {0x6f913c0, 0xc013b47ba0}, 0xc025670a20, 0xc0011f7380, 0xac756a0, 0x0)
external/org_golang_google_grpc/server.go:1659 +0x1265
google.golang.org/grpc.(*Server).handleStream(0xc0037921e0, {0x6f913c0, 0xc013b47ba0}, 0xc025670a20, 0x0)
external/org_golang_google_grpc/server.go:1739 +0x987
google.golang.org/grpc.(*Server).serveStreams.func1.1()
external/org_golang_google_grpc/server.go:970 +0xaf
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 104084067
external/org_golang_google_grpc/server.go:981 +0x133

goroutine 106985374 [runnable]:
github.com/sasha-s/go-deadlock.lock.func1()
external/com_github_sasha_s_go_deadlock/deadlock.go:182
created by github.com/sasha-s/go-deadlock.lock in goroutine 26772793
external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x151

goroutine 106899694 [runnable]:
github.com/sasha-s/go-deadlock.lock.func1()
external/com_github_sasha_s_go_deadlock/deadlock.go:182
created by github.com/sasha-s/go-deadlock.lock in goroutine 327647
external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x151

goroutine 106984194 [runnable]:
github.com/sasha-s/go-deadlock.lock.func1()
external/com_github_sasha_s_go_deadlock/deadlock.go:182
created by github.com/sasha-s/go-deadlock.lock in goroutine 26772793
external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x151

goroutine 106903207 [runnable]:
github.com/sasha-s/go-deadlock.lock.func1()
external/com_github_sasha_s_go_deadlock/deadlock.go:182
created by github.com/sasha-s/go-deadlock.lock in goroutine 71
external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x151

goroutine 106988280 [runnable]:
github.com/sasha-s/go-deadlock.lock.func1()
external/com_github_sasha_s_go_deadlock/deadlock.go:182
created by github.com/sasha-s/go-deadlock.lock in goroutine 105358109
external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x151
```

Parameters:
- attempt=1
- deadlock=true
- run=3
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #172692 util/startup: TestStartupFailureRandomRange failed [C-test-failure O-robot branch-master release-blocker]

/cc @cockroachlabs/unowned

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestStartupFailureRandomRange.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66047

Contributor guide

Open the contributing guide

Research direction

Start with util/startup.TestStartupFailureRandomRange and reproduce the release-26.2 failure using the linked invocation details. Read the referenced internal guide and inspect the goroutine stack, including pkg/rpc/auth.go:192 and pkg/util/stop/stopper.go:358; done means the intermittent failure is explained and the relevant test or startup behavior is verified across the reported branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.