cockroachdb / cockroachdb/cockroach

crosscluster/physical: TestIngestSpanConfigsFullScan failed

Open
#175,119 0 comments 0 reactions 0 assignees View on GitHub
branch-staging-v25.4.16 C-test-failure O-robot release-blocker
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

crosscluster/physical.TestIngestSpanConfigsFullScan [failed](https://mesolite.cluster.engflow.com/invocations/default/62c82f29-a7d2-4286-b985-c719f1445695?testReportRun=1&testReportShard=14&testReportAttempt=1#targets-Ly9wa2cvY3Jvc3NjbHVzdGVyL3BoeXNpY2FsOnBoeXNpY2FsX3Rlc3Q=) on staging-v25.4.16 @ [65ac8a6bd3d10290de3f25297aaa6d1c3de3c369](https://github.com/cockroachdb/cockroach/commits/65ac8a6bd3d10290de3f25297aaa6d1c3de3c369):

```
crypto/tls.(*Conn).readRecordOrCCS(0xc0094c0708, 0x0)
GOROOT/src/crypto/tls/conn.go:629 +0x3cf
crypto/tls.(*Conn).readRecord(...)
GOROOT/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc0094c0708, {0xc0096a0000, 0x8000, 0xc006c75cf0?})
GOROOT/src/crypto/tls/conn.go:1385 +0x150
bufio.(*Reader).Read(0xc0095d3ce0, {0xc00302e2e0, 0x9, 0x0?})
GOROOT/src/bufio/bufio.go:241 +0x197
io.ReadAtLeast({0x88090e0, 0xc0095d3ce0}, {0xc00302e2e0, 0x9, 0x9}, 0x9)
GOROOT/src/io/io.go:335 +0x90
io.ReadFull(...)
GOROOT/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc00302e2e0, 0x9, 0xc0093d2bd0?}, {0x88090e0?, 0xc0095d3ce0?})
external/org_golang_x_net/http2/frame.go:242 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00302e2a0)
external/org_golang_x_net/http2/frame.go:506 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0095b4488, 0xc0095b75e0)
external/org_golang_google_grpc/internal/transport/http2_client.go:1595 +0x226
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 7365
external/org_golang_google_grpc/internal/transport/http2_client.go:397 +0x1d33

goroutine 7341 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0092ab040)
external/org_golang_google_grpc/internal/transport/http2_server.go:1150 +0x1f3
created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 7336
external/org_golang_google_grpc/internal/transport/http2_server.go:339 +0x1a8e

goroutine 7339 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00944afa0, 0x1)
external/org_golang_google_grpc/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0094fefc0)
external/org_golang_google_grpc/internal/transport/controlbuf.go:552 +0x7b
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
external/org_golang_google_grpc/internal/transport/http2_client.go:451 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 7334
external/org_golang_google_grpc/internal/transport/http2_client.go:449 +0x22db

goroutine 87160 [select]:
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x886fa88, 0xc006a12960})
pkg/sql/stats/automatic_stats.go:625 +0x1b5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x886fa88, 0xc006a12960})
pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 1965
pkg/util/stop/stopper.go:445 +0x137

goroutine 78122 [select]:
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
external/com_github_cockroachdb_pebble/vfs/disk_health.go:254 +0xb3
created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 508
external/com_github_cockroachdb_pebble/vfs/disk_health.go:249 +0x58
```

Parameters:
- attempt=1
- run=1
- shard=14
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/unowned

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestIngestSpanConfigsFullScan.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with the linked Mesolite test report and the crosscluster/physical.TestIngestSpanConfigsFullScan failure on staging-v25.4.16; inspect the reported Go TLS/gRPC goroutine stack and the referenced commit. Reproduce the failure and identify its cause, with the test passing reliably as the done condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.