cockroachdb / cockroachdb/cockroach

ccl/multiregionccl: TestRegionLivenessProberForLeases failed

Open
#172,845 4 comments 0 reactions 0 assignees View on GitHub
branch-master C-test-failure O-robot T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

ccl/multiregionccl.TestRegionLivenessProberForLeases [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21440922?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21440922?buildTab=artifacts#/) on master @ [dbfee5672e06e6a455a79d70f7ad46a47f6707ac](https://github.com/cockroachdb/cockroach/commits/dbfee5672e06e6a455a79d70f7ad46a47f6707ac):

```

goroutine 44211 [select, 14 minutes, "tags":"n1,rnode=2,raddr=127.0.0.1:37353,class=default,rpc", "workload.id":"RPC_HEARTBEAT"]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x31a1230daf40, {0x712d910, 0x31a1234b74f0})
external/org_golang_google_grpc/internal/grpcsync/callback_serializer.go:83 +0xc0
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 44210
external/org_golang_google_grpc/internal/grpcsync/callback_serializer.go:55 +0x10c

goroutine 39066 [select, 14 minutes]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).runPeriodicSnapshotsLoop(0x31a0fa9e5ef8?, 0x31a0f9698000, 0x31a1023aa700, 0x31a13269b490, 0x70cc620?)
pkg/util/tracing/tracer_snapshots.go:279 +0xbc
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).PeriodicSnapshotsLoop(0x31a11a5c4b40, 0x31a0f9698000, 0x31a13269b490)
pkg/util/tracing/tracer_snapshots.go:245 +0xb0
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func1({0x31a126322540?, 0x71306e8?})
pkg/server/server_sql.go:575 +0x44
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x71306e8, 0x31a1224866e0})
pkg/util/stop/stopper.go:447 +0x7c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 39410
pkg/util/stop/stopper.go:445 +0xe0

goroutine 44827 [select, 14 minutes, "n":"3", "replicate":""]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x712d948, 0x31a12124b620})
pkg/kv/kvserver/queue.go:1131 +0xbc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x712d948, 0x31a12124b620})
pkg/util/stop/stopper.go:447 +0x7c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 44590
pkg/util/stop/stopper.go:445 +0xe0

goroutine 44829 [select, "n":"3", "raftlog":""]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x712d948, 0x31a12124b6e0})
pkg/kv/kvserver/queue.go:1131 +0xbc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x712d948, 0x31a12124b6e0})
pkg/util/stop/stopper.go:447 +0x7c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 44590
pkg/util/stop/stopper.go:445 +0xe0

goroutine 44830 [select, 14 minutes, "n":"3", "raftsnapshot":""]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x712d948, 0x31a12124b710})
pkg/kv/kvserver/queue.go:1131 +0xbc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x712d948, 0x31a12124b710})
pkg/util/stop/stopper.go:447 +0x7c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 44590
pkg/util/stop/stopper.go:445 +0xe0

goroutine 44828 [select, 10 minutes, "n":"3", "replicaGC":""]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x712d948, 0x31a12124b650})
pkg/kv/kvserver/queue.go:1131 +0xbc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x712d948, 0x31a12124b650})
pkg/util/stop/stopper.go:447 +0x7c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 44590
pkg/util/stop/stopper.go:445 +0xe0
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestRegionLivenessProberForLeases.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66111

Contributor guide

Open the contributing guide

Research direction

Start with the TeamCity failure log and artifacts for ccl/multiregionccl.TestRegionLivenessProberForLeases at commit dbfee5672e06e6a455a79d70f7ad46a47f6707ac. Inspect the reported goroutine stacks and the test entry point to determine why the test hangs or fails. Done means identifying the failure cause, addressing it, and confirming the test passes reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.