kubernetes / kubernetes/cloud-provider-openstack

[occm]: introduce readiness gates for pods

Open
#2,148 31 comments 4 reactions 0 assignees View on GitHub
kind/feature
Dominant language
Go
Stars
720
Forks
687
Avg merge
1d 14h
Merged PRs (30d)
51

Description

**Is this a BUG REPORT or FEATURE REQUEST?**:

/kind feature

**What happened**:

Current LoadBalancer service implementation has a flaw: if during the deployment update a node's network is broken, but kubelet advertises that the pod is up and ready, the new deployment can cause an outage by shutting down old healthy pods.

**What you expected to happen**:

There should be a way to monitor pod's network readiness outside the k8s cluster using pods' [readiness gates](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate). See more details in this video: https://www.youtube.com/watch?v=Vw9GmSeomFg&t=289s

**Anything else we need to know?**:

In #1720 PR the `externalTrafficPolicy: Local` was introduced, which adds kube-proxy based monitors to LB pool members. We can keep this logic for OCCM configured without a router.

However for OCCM with a router configured, we can use service's endpoints instead of node ports and according to loadbalancer's member healthchecks patch the pods' readiness gates accordingly. This approach would increase a deployment update time (because of the LB healthchecks latency), but from other side this will increase the overall deployment accessibility. Additional set of advantages: more even traffic distribution between pods, and pod-based traffic affinity.

See also https://cloud.google.com/kubernetes-engine/docs/concepts/container-native-load-balancing#pod_readiness and https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/pod_readiness_gate/ descriptions.
Unlike GCE and AWS dedicated ingress/LB controllers, I'd like to implement readiness gates feature directly in the OCCM controller for seamless feature toggle, especially for existing deployments.

Feel free to provide your suggestions or objections.

cc @databus23 @jichenjc @zetaab

Contributor guide

Open the contributing guide

Research direction

Start by reading the OCCM controller logic described alongside PR #1720, then review Kubernetes pod readiness gates and how service endpoints and load-balancer member health checks relate. Clarify the feature scope and toggle behavior before implementing; done means pod readiness gates reflect the configured load balancer health state without regressing the no-router behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.