apache / apache/airflow

pgbouncer prestop command is not graceful enough

Open
#69,571 1 comment 0 reactions 0 assignees View on GitHub
area:helm-chart kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Helm chart

### Apache Airflow version

2.11.0(airflow version is not applicable to this issue)

### What happened and how to reproduce it?

To reproduce:
- perform a rolling restart of the pgbouncer deployment(with multiple replicas) in an airflow cluster and kubernetes cluster under heavy load
- observe scheduler, dag-processor, and worker pods
- some pods will see restarts with a SQL connection error like "No route to host" or "Connection refused"

### What you think should happen instead?

The prestop command used for pgbouncer in the helm chart(shown below) is problematic:
```
containerLifecycleHooks:
preStop:
exec:
# Allow existing queries clients to complete within 120 seconds
command: ["/bin/sh", "-c", "killall -INT pgbouncer && sleep 120"]
```
Although `killall -INT pgbouncer` does in fact trigger a safe shutdown as expected, it exits as soon as in-flight queries drain. Once this occurs, the liveness probe terminates the pod regardless of whether the `sleep 120` has finished or not. In the event of an idle pgbouncer pod or one with no long-running queries, this happens in ~1 second or less. The container therefore stops accepting connections before the EndpointSlice/kube-proxy updates triggered by the pod entering Terminating have propagated, so new connections routed to the still-registered (but dead) pod fail with connection errors.

I've fixed this by putting a short sleep command in front of `killall -INT pgbouncer` like so:
```
containerLifecycleHooks:
preStop:
exec:
# Allow existing queries clients to complete within 120 seconds
command: ["/bin/sh", "-c", "sleep 10 && killall -INT pgbouncer && sleep 20"]
```
This allows the pod to transition to a `terminating` state while the pgbouncer container continues to run, which grants kubernetes more time to remove the target from the service endpoint and update the loadbalancer and kube-proxy rules. Now, if a query gets sent to the terminating pod before its been removed from the service endpoint fully, it is less likely to experience a connection failure because pgbouncer is still running and will still accept the query.

```
➜ ~ kubectl get endpointslice -o json -n airflow airflow--pgbouncer-n57hm | grep '"terminating": true' -B2
--
"ready": true,
"serving": true,
"terminating": true
```

Also, I made the post sleep command `20` becuase there is no `TerminationGracePeriod` defined for the pgbouncer deployment in the chart, so it ends up defaulting to 30s. Basically, the `sleep 120` doesn't really do what it says and is therefore misleading to users of the chart.

### Operating System

Amazon Linux 2023(fedora)

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

1.22.0 (latest released)

### Kubernetes Version

1.35

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by locating the Helm chart template for the pgbouncer deployment and its containerLifecycleHooks.preStop command. Review the interaction between the hook, liveness probe, EndpointSlice updates, and the deployment's termination grace period. Done means rolling restarts under load no longer produce connection errors and the configured shutdown timing is accurate.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.