Packets sent to primary when both primary and secondary destinations are unhealthy
- Dominant language
- C
- Stars
- 2.5k
- Forks
- 239
- PR merge metrics
- No merged PRs in 30d
Description
While doing some testing, @ravisinghsfbay and @robloxrob and I ran into a case where glb-director forwards ingress packets to a known unhealthy destination. We're interested in submitting a patch to fix this case, but first I'm hoping to get a quick confirmation that the root-cause analysis below is correct.
The test setup was:
* Set up glb-director in front of a pool of four backend servers.
* Send some requests through glb-director to verify that the load balancing is working.
* Then turn off two of the backend servers so they stop passing health checks.
Expected result:
* glb-director forwards subsequent TCP connections to one of the healthy servers.
Observed result:
* glb-director forwards subsequent TCP connections to one of the unhealthy servers. The metadata in the GUE header indicates that the other unhealthy server is that server's secondary.
I verified that glb-healthcheck detected the failed healthchecks and generated a `forwarding_table.checked.bin` that properly identified the affected servers as state=1 and health=0.
From a quick read through the code, I think the problem is in the construction of the forwarding table in `glb-director/cli/main.c`. There's some code checks for the case where the primary is unhealthy and the secondary is healthy. But if _both_ the primary and the secondary are unhealthy, that entry in the forwarding table ends up with two unhealthy destinations in it. In the data plane, if an incoming packet hashes to that entry in the forwarding table, glb-director then sends it to the (unhealthy) primary, with a GUE wrapper that lists the (unhealthy) secondary.
Assuming that analysis is correct, the first idea that springs to mind is to increase the number of destination IPs stored in each routing table slot from 2 to a configurable _N_. That wouldn't prevent the problem altogether, but it would reduce the probability of the problem as _N_ increased.
Contributor guide
Assessment
This issue has not been assessed yet.