cockroachdb / cockroachdb/cockroach

streamclient: NewPartitionedStreamClient` can hang under network partition

Open
#134,676 3 comments 0 reactions 0 assignees View on GitHub
branch-master C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In a updated c2c/disconnect roachtest, the test will fail because the `setupPGXConfig` call in `NewPartitionStreamClient` can hang for 10+ minutes. The failures can be especially harmful if the coordinator node disconnects, as there are more client connections it needs to open.

https://github.com/msbutler/cockroach/blob/butler-deflake-ldr-partition/pkg/ccl/crosscluster/streamclient/pgconn.go#L36

Here's an example stack. I've observed this hanging behavior during distsql processor start and on the coordinator, which implies the hanging behavior is non-deterministic: occasionally the `NewPartionStreamClient` call will error quickly.
```
goroutine 26875 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x70e1db8f8ec8, 0x77)
GOROOT/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc001b55200?, 0x0?, 0x0)
GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitWrite(...)
GOROOT/src/internal/poll/fd_poll_runtime.go:93
internal/poll.(*FD).WaitWrite(...)
GOROOT/src/internal/poll/fd_unix.go:683
net.(*netFD).connect(0xc001b55200, {0x873e660, 0xc00181bd40}, {0xc019800bf0?, 0x4707db?}, {0x86e24e0?, 0xc0026332a0?})
GOROOT/src/net/fd_unix.go:141 +0x70b
net.(*netFD).dial(0xc001b55200, {0x873e660, 0xc00181bd40}, {0x8775fa8?, 0x0?}, {0x8775fa8, 0xc00186d770}, 0xc00169e4a0?)
GOROOT/src/net/sock_posix.go:124 +0x3bc
net.socket({0x873e660, 0xc00181bd40}, {0x6e51eb3, 0x3}, 0x2, 0x1, 0xc00186d740?, 0x0, {0x8775fa8, 0x0}, ...)
GOROOT/src/net/sock_posix.go:70 +0x29b
net.internetSocket({0x873e660, 0xc00181bd40}, {0x6e51eb3, 0x3}, {0x8775fa8, 0x0}, {0x8775fa8, 0xc00186d770}, 0x1, 0x0, ...)
GOROOT/src/net/ipsock_posix.go:154 +0xf8
net.(*sysDialer).doDialTCPProto(0xc0103aeb40, {0x873e660, 0xc00181bd40}, 0x0, 0xc00186d770, 0x0)
GOROOT/src/net/tcpsock_posix.go:85 +0xec
net.(*sysDialer).doDialTCP(...)
GOROOT/src/net/tcpsock_posix.go:75
net.(*sysDialer).dialTCP(0xc019800e50?, {0x873e660?, 0xc00181bd40?}, 0xc019800e68?, 0x50c7a4?)
GOROOT/src/net/tcpsock_posix.go:71 +0x65
net.(*sysDialer).dialSingle(0xc0103aeb40, {0x873e660, 0xc00181bd40}, {0x86fe428, 0xc00186d770})
GOROOT/src/net/dial.go:651 +0x27d
net.(*sysDialer).dialSerial(0xc0103aeb40, {0x873e660, 0xc00181bd40}, {0xc002779420?, 0x1, 0xc002779420?})
GOROOT/src/net/dial.go:616 +0x24e
net.(*sysDialer).dialParallel(0xc002779410?, {0x873e660?, 0xc00181bd40?}, {0xc002779420?, 0xc00181bd40?, 0x6e54be3?}, {0x0?, 0x6e51eb3?, 0xc0120967a5?})
GOROOT/src/net/dial.go:517 +0x3b4
net.(*Dialer).DialContext(0xc00dd30690, {0x873e660, 0xc00181bd40}, {0x6e51eb3, 0x3}, {0xc012096798, 0x12})
GOROOT/src/net/dial.go:508 +0x69a
github.com/jackc/pgconn.connect({0x873e660, 0xc00181bd40}, 0xc0053d58c0, 0xc00169ef20, 0x0)
external/com_github_jackc_pgconn/pgconn.go:267 +0x19d
github.com/jackc/pgconn.ConnectConfig({0x873e660, 0xc00181bd40}, 0xc0053d58c0)
external/com_github_jackc_pgconn/pgconn.go:168 +0x451
github.com/jackc/pgx/v4.connect({0x873e660, 0xc00181bd40}, 0xc0053d5440)
external/com_github_jackc_pgx_v4/conn.go:222 +0x365
github.com/jackc/pgx/v4.ConnectConfig(...)
external/com_github_jackc_pgx_v4/conn.go:113
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/streamclient.newPGConnForClient({0x873e660, 0xc00181bd40}, 0x0?, 0x15?)
pkg/ccl/crosscluster/streamclient/pgconn.go:40 +0x45
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/streamclient.NewPartitionedStreamClient({0x873e660, 0xc00181bd40}, 0xc009ba3dd0, {0xc004d81e90, 0x2, 0xc004d819f8?})
pkg/ccl/crosscluster/streamclient/partitioned_stream_client.go:50 +0xab
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/streamclient.NewStreamClient({0x873e660, 0xc00181bd40}, {0xc016eb6600?, 0x873e660?}, {0x873e270, 0xc004d12990}, {0xc004d81e90, 0x2, 0x2})
pkg/ccl/crosscluster/streamclient/client.go:231 +0x18d
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/physical.(*streamIngestionProcessor).Start(0xc00de72008, {0x873e660, 0xc0163c8510})
pkg/ccl/crosscluster/physical/stream_ingestion_processor.go:440 +0x9bf
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0xc00de72008, {0x873e660?, 0xc0163c8510?}, {0x8713398, 0xc00ddec008})
pkg/sql/execinfra/processorsbase.go:726 +0x4a
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal.func1(0xc00c751fa0?)
pkg/sql/flowinfra/flow.go:508 +0x5d
created by github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal in goroutine 26867
pkg/sql/flowinfra/flow.go:507 +0x405
```

Jira issue: CRDB-44196

Contributor guide

Open the contributing guide

Research direction

Start in pkg/ccl/crosscluster/streamclient/pgconn.go around newPGConnForClient and follow its call from NewPartitionedStreamClient in partitioned_stream_client.go. Reproduce the updated c2c/disconnect roachtest and inspect the supplied stack showing pgx connection setup blocked in net.Dial. Done means the client setup returns promptly rather than hanging for 10+ minutes during a network partition.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.