grpc / grpc/grpc-java

PickFirstLeafLoadBalancer may get stuck in Idle on backend address change

Open
#12,395 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

A bug was found in the new pickfirst balancer in gRPC Go which can cause the balancer to get stuck in `IDLE` state: https://github.com/grpc/grpc-go/pull/8615.

The implementation of the pickfirst LB in Java is similar, so it may suffer from the same issue. The order of events that lead to the bug is as follows:
1. Existing connection breaks, the balancer [requests re-resolution and reports IDLE](https://github.com/grpc/grpc-java/blob/7d5749f39d2d1665ad9a2fc5d6d7adf1533effdc/core/src/main/java/io/grpc/internal/PickFirstLeafLoadBalancer.java#L253-L255). PF [updates the channel state to IDLE](https://github.com/grpc/grpc-java/blob/7d5749f39d2d1665ad9a2fc5d6d7adf1533effdc/core/src/main/java/io/grpc/internal/PickFirstLeafLoadBalancer.java#L280-L285) with an `Idle picker`.
1. An RPC is made, [triggering the balancer to exit idle](https://sourcegraph.com/github.com/grpc/grpc-java@7d5749f39d2d1665ad9a2fc5d6d7adf1533effdc/-/blob/core/src/main/java/io/grpc/internal/PickFirstLeafLoadBalancer.java?L628-630) through the picker. The balancer attempts to re-connect the failed subchannel.
1. The resolver produces a new endpoint list, removing the endpoint used by the existing subchannel. PF removes the existing subchannel. Since the balancer didn't update the channel state to CONNECTING yet, pickfirst thinks that it's still in IDLE and doesn't start connecting to the new endpoints.
1. New RPC requests trigger the idle picker, but it's a no-op since it only [triggers the balancer's ExitIdle method once](https://github.com/grpc/grpc-java/blob/7d5749f39d2d1665ad9a2fc5d6d7adf1533effdc/core/src/main/java/io/grpc/internal/PickFirstLeafLoadBalancer.java#L628).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.