cockroachdb / cockroachdb/cockroach
streamclient: NewPartitionedStreamClient` can hang under network partition
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In a updated c2c/disconnect roachtest, the test will fail because the `setupPGXConfig` call in `NewPartitionStreamClient` can hang for 10+ minutes. The failures can be especially harmful if the coordinator node disconnects, as there are more client connections it needs to open.
https://github.com/msbutler/cockroach/blob/butler-deflake-ldr-partition/pkg/ccl/crosscluster/streamclient/pgconn.go#L36
Here's an example stack. I've observed this hanging behavior during distsql processor start and on the coordinator, which implies the hanging behavior is non-deterministic: occasionally the `NewPartionStreamClient` call will error quickly.
```
goroutine 26875 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x70e1db8f8ec8, 0x77)
GOROOT/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc001b55200?, 0x0?, 0x0)
GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitWrite(...)
GOROOT/src/internal/poll/fd_poll_runtime.go:93
internal/poll.(*FD).WaitWrite(...)
GOROOT/src/internal/poll/fd_unix.go:683
net.(*netFD).connect(0xc001b55200, {0x873e660, 0xc00181bd40}, {0xc019800bf0?, 0x4707db?}, {0x86e24e0?, 0xc0026332a0?})
GOROOT/src/net/fd_unix.go:141 +0x70b
net.(*netFD).dial(0xc001b55200, {0x873e660, 0xc00181bd40}, {0x8775fa8?, 0x0?}, {0x8775fa8, 0xc00186d770}, 0xc00169e4a0?)
GOROOT/src/net/sock_posix.go:124 +0x3bc
net.socket({0x873e660, 0xc00181bd40}, {0x6e51eb3, 0x3}, 0x2, 0x1, 0xc00186d740?, 0x0, {0x8775fa8, 0x0}, ...)
GOROOT/src/net/sock_posix.go:70 +0x29b
net.internetSocket({0x873e660, 0xc00181bd40}, {0x6e51eb3, 0x3}, {0x8775fa8, 0x0}, {0x8775fa8, 0xc00186d770}, 0x1, 0x0, ...)
GOROOT/src/net/ipsock_posix.go:154 +0xf8
net.(*sysDialer).doDialTCPProto(0xc0103aeb40, {0x873e660, 0xc00181bd40}, 0x0, 0xc00186d770, 0x0)
GOROOT/src/net/tcpsock_posix.go:85 +0xec
net.(*sysDialer).doDialTCP(...)
GOROOT/src/net/tcpsock_posix.go:75
net.(*sysDialer).dialTCP(0xc019800e50?, {0x873e660?, 0xc00181bd40?}, 0xc019800e68?, 0x50c7a4?)
GOROOT/src/net/tcpsock_posix.go:71 +0x65
net.(*sysDialer).dialSingle(0xc0103aeb40, {0x873e660, 0xc00181bd40}, {0x86fe428, 0xc00186d770})
GOROOT/src/net/dial.go:651 +0x27d
net.(*sysDialer).dialSerial(0xc0103aeb40, {0x873e660, 0xc00181bd40}, {0xc002779420?, 0x1, 0xc002779420?})
GOROOT/src/net/dial.go:616 +0x24e
net.(*sysDialer).dialParallel(0xc002779410?, {0x873e660?, 0xc00181bd40?}, {0xc002779420?, 0xc00181bd40?, 0x6e54be3?}, {0x0?, 0x6e51eb3?, 0xc0120967a5?})
GOROOT/src/net/dial.go:517 +0x3b4
net.(*Dialer).DialContext(0xc00dd30690, {0x873e660, 0xc00181bd40}, {0x6e51eb3, 0x3}, {0xc012096798, 0x12})
GOROOT/src/net/dial.go:508 +0x69a
github.com/jackc/pgconn.connect({0x873e660, 0xc00181bd40}, 0xc0053d58c0, 0xc00169ef20, 0x0)
external/com_github_jackc_pgconn/pgconn.go:267 +0x19d
github.com/jackc/pgconn.ConnectConfig({0x873e660, 0xc00181bd40}, 0xc0053d58c0)
external/com_github_jackc_pgconn/pgconn.go:168 +0x451
github.com/jackc/pgx/v4.connect({0x873e660, 0xc00181bd40}, 0xc0053d5440)
external/com_github_jackc_pgx_v4/conn.go:222 +0x365
github.com/jackc/pgx/v4.ConnectConfig(...)
external/com_github_jackc_pgx_v4/conn.go:113
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/streamclient.newPGConnForClient({0x873e660, 0xc00181bd40}, 0x0?, 0x15?)
pkg/ccl/crosscluster/streamclient/pgconn.go:40 +0x45
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/streamclient.NewPartitionedStreamClient({0x873e660, 0xc00181bd40}, 0xc009ba3dd0, {0xc004d81e90, 0x2, 0xc004d819f8?})
pkg/ccl/crosscluster/streamclient/partitioned_stream_client.go:50 +0xab
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/streamclient.NewStreamClient({0x873e660, 0xc00181bd40}, {0xc016eb6600?, 0x873e660?}, {0x873e270, 0xc004d12990}, {0xc004d81e90, 0x2, 0x2})
pkg/ccl/crosscluster/streamclient/client.go:231 +0x18d
github.com/cockroachdb/cockroach/pkg/ccl/crosscluster/physical.(*streamIngestionProcessor).Start(0xc00de72008, {0x873e660, 0xc0163c8510})
pkg/ccl/crosscluster/physical/stream_ingestion_processor.go:440 +0x9bf
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0xc00de72008, {0x873e660?, 0xc0163c8510?}, {0x8713398, 0xc00ddec008})
pkg/sql/execinfra/processorsbase.go:726 +0x4a
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal.func1(0xc00c751fa0?)
pkg/sql/flowinfra/flow.go:508 +0x5d
created by github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal in goroutine 26867
pkg/sql/flowinfra/flow.go:507 +0x405
```
Jira issue: CRDB-44196
Contributor guide
Research direction
Start in pkg/ccl/crosscluster/streamclient/pgconn.go around newPGConnForClient and follow its call from NewPartitionedStreamClient in partitioned_stream_client.go. Reproduce the updated c2c/disconnect roachtest and inspect the supplied stack showing pgx connection setup blocked in net.Dial. Done means the client setup returns promptly rather than hanging for 10+ minutes during a network partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- databases, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100