kv/kvnemesis: TestKVNemesisMultiNode_Partition_Liveness failed [DistSender livelock on partition]

kv/kvnemesis.TestKVNemesisMultiNode_Partition_Liveness [failed](https://mesolite.cluster.engflow.com/invocations/default/a52286ee-7a8d-420c-a276-393eaedbfb40?testReportRun=10&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZuZW1lc2lzOmt2bmVtZXNpc190ZXN0) on master @ [bf01372d7ca0f2e6ad6dfa107d679182aee173dd](https://github.com/cockroachdb/cockroach/commits/bf01372d7ca0f2e6ad6dfa107d679182aee173dd):

```
io.ReadAtLeast({0x6c56700, 0xd32dc4ea600}, {0xd32dc4ea670, 0x5, 0x5}, 0x5)
GOROOT/src/io/io.go:335 +0x8e
io.ReadFull(...)
GOROOT/src/io/io.go:354
google.golang.org/grpc/internal/transport.(*Stream).Read(0xd32beb3efc0, {0xd32dc4ea670, 0x5, 0x5})
external/org_golang_google_grpc/internal/transport/transport.go:498 +0x96
google.golang.org/grpc.(*parser).recvMsg(0xd32dc4ea660, 0x7fffffff)
external/org_golang_google_grpc/rpc_util.go:600 +0x43
google.golang.org/grpc.recvAndDecompress(0x7fffffff?, 0xd32beb3efc0, {0x0, 0x0}, 0x7fffffff, 0x0, {0x6c728f0, 0xcadea00})
external/org_golang_google_grpc/rpc_util.go:730 +0x5a
google.golang.org/grpc.recv(0xd32dc4ea660, {0x7f6ece280e68, 0xd32a71769d0}, 0xd32beb3efc0, {0x0, 0x0}, {0x6524540, 0xd32c1336fc0}, 0x7fffffff, 0x0, ...)
external/org_golang_google_grpc/rpc_util.go:796 +0xa7
google.golang.org/grpc.(*serverStream).RecvMsg(0xd32c8ea74a0, {0x6524540, 0xd32c1336fc0})
external/org_golang_google_grpc/stream.go:1712 +0x13b
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*internalMuxRangeFeedServer).Recv(0xd32cb470a10)
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12624 +0x46
github.com/cockroachdb/cockroach/pkg/server.(*Node).muxRangeFeed(0xd32cc765c08, {0x7f6ecb767650, 0xd32cb470a10})
pkg/server/node.go:2349 +0x584
github.com/cockroachdb/cockroach/pkg/server.(*Node).MuxRangeFeed(0xc34a030?, {0x6ce80b8?, 0xd32cb470a10?})
pkg/server/node.go:2304 +0x3e
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_MuxRangeFeed_Handler({0x66d8b60?, 0xd32cc765c08}, {0x6cd1140, 0xd32c7a31120})
bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:12605 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func14({0x66d8b60, 0xd32cc765c08}, {0x6cd27a8, 0xd32c8ea74a0}, 0xd32eafba048, 0x6c06b78)
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:137 +0x399
google.golang.org/grpc.getChainStreamHandler.func1({0x66d8b60?, 0xd32cc765c08?}, {0x6cd27a8?, 0xd32c8ea74a0?})
external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x66d8b60, 0xd32cc765c08}, {0x6cd27a8, 0xd32c8ea74a0}, 0xd32eafba048?, 0xd32c6224f40)
pkg/rpc/context.go:206 +0x63
google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1({0x66d8b60, 0xd32cc765c08}, {0x6cd27a8, 0xd32c8ea74a0})
external/org_golang_google_grpc/server.go:1504 +0xa8
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xd32bb102000, {{0x1}, {0x6cf5b00, 0xd32d9fe8190}}, 0x0}, {0x66d8b60, 0xd32cc765c08}, {0x6cd27a8, 0xd32c8ea74a0}, 0xd32eafba048, ...)
pkg/rpc/auth.go:204 +0x3ba
google.golang.org/grpc.getChainStreamHandler.func1({0x66d8b60?, 0xd32cc765c08?}, {0x6cd27a8?, 0xd32c8ea74a0?})
external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x0?, 0x6c958f0?})
pkg/rpc/context.go:163 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xd32d20f8900, {0x6c958f0, 0xd32dc4ea6c0}, {0xd32edd72d50, 0x28}, 0xd32daedaaa8)
pkg/util/stop/stopper.go:358 +0x138
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x66d8b60, 0xd32cc765c08}, {0x6cd27a8, 0xd32c8ea74a0}, 0xd32eafba048, 0xd32c6224ec0)
pkg/rpc/context.go:162 +0xcc
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x66d8b60, 0xd32cc765c08}, {0x6cd27a8, 0xd32c8ea74a0}, 0xd32eafba048, 0xd32cb470a00?)
external/org_golang_google_grpc/server.go:1495 +0x7c
google.golang.org/grpc.(*Server).processStreamingRPC(0xd32c4753a40, {0x6cf6880, 0xd32bb59bd40}, 0xd32beb3efc0, 0xd32d70b77d0, 0xc43b660, 0x0)
external/org_golang_google_grpc/server.go:1659 +0x1251
google.golang.org/grpc.(*Server).handleStream(0xd32c4753a40, {0x6cf6880, 0xd32bb59bd40}, 0xd32beb3efc0, 0x0)
external/org_golang_google_grpc/server.go:1739 +0x987
google.golang.org/grpc.(*Server).serveStreams.func1.1()
external/org_golang_google_grpc/server.go:970 +0xaf
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 129093
external/org_golang_google_grpc/server.go:981 +0x133
```

Parameters:
- attempt=1
- run=10
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/kv-triage

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67707