goauthentik / goauthentik/authentik

PostgresChannelLayer exhausts PgBouncer connection pool under normal load, causing cascading Authentik outages

Open
#22,412 1 comment 0 reactions 0 assignees View on GitHub
bug bug/confirmed
Dominant language
Python
Stars
25.6k
Forks
2k
Avg merge
1d 2h
Merged PRs (30d)
651

Description

### Describe the bug

## Summary

The `PostgresChannelLayer` introduced as the default channel layer in Authentik 2026.2.x creates a sustained, high baseline of simultaneous PostgreSQL connections. Under normal operating conditions — with a modest number of WebSocket subscribers (admin UI sessions, outpost connections, flow consumers) — this baseline alone approaches or exceeds typical PgBouncer pool ceilings. Any additional database load (scheduled task bursts, LDAP authentication, gunicorn request traffic) tips the pool into exhaustion, causing a cascade where Authentik becomes completely unresponsive.

This is a silent failure mode: PostgreSQL itself reports healthy throughout; the bottleneck is entirely at the PgBouncer connection pool layer, making it non-obvious to diagnose.

### How to reproduce

This issue is emergent under normal load rather than a discrete click-path reproduction. The following conditions reliably trigger it:

**Environment:**
- Authentik 2026.2.2, Docker Compose deployment
- PgBouncer in transaction pooling mode (`edoburu/pgbouncer:v1.25.1-p0`)
- `default_pool_size` at or below 100 (default shipping value is 35)
- PostgreSQL `max_connections = 500`
**Steps:**

1. Start Authentik stack with default PgBouncer `default_pool_size = 35`
2. Allow normal operation for a few minutes (admin UI open in one or more tabs, at least one outpost connected)
3. Query pg_stat_activity immediately:
```sql
SELECT count(*) AS channels_conns
FROM pg_stat_activity
WHERE query LIKE '%django_channels_postgres_groupchannel%';
```

You will observe 62–78 connections already consumed by the channel layer at baseline — exceeding the default pool ceiling of 40 (35 default + 5 reserve) before any user traffic hits.

4. Within the first cron coincidence window (typically within 30 minutes), Authentik's task scheduler fires multiple scheduled tasks simultaneously. The resulting PostgreSQL NOTIFY burst wakes all Dramatiq consumer threads at once, each requiring a pool slot. With the channel layer already consuming the majority of available slots, `cl_waiting` climbs rapidly and PgBouncer begins killing connections after `query_wait_timeout = 120s`.
5. Authentik enters a crash loop. The LDAP outpost goes unhealthy (bind requests timing out at 240,000–260,000ms). The web UI shows "server is starting" indefinitely. Manual intervention is required to recover.

### Expected behavior

The default channel layer configuration should not consume enough database connections under normal idle conditions to make Authentik unable to handle any additional load. If `PostgresChannelLayer` requires a large number of persistent connections by design, the following should be true:

- Deployment documentation should specify minimum recommended PgBouncer `pool_size` as a function of expected WebSocket subscriber count
- Authentik should log a startup warning if the detected pool capacity is insufficient for the configured channel layer
- The upgrade notes from 2026.1.x → 2026.2.x should document the channel layer change and its operational implications, particularly for PgBouncer deployments

### Screenshots

*(Attached: PgBouncer connection timeline chart showing cl_active climbing from baseline ~20 to 147+ and cl_waiting reaching 246, with pool ceiling at 80. Connection age distribution showing ~80 connections in the >5min bracket at onset — consistent with long-held channel layer connections.)*

Image

### Additional context

**Connection math at baseline (pool_size = 80, reserve = 5 → ceiling = 85):**

| Consumer | Connections |
|---|---|
| `django_channels_postgres` baseline | 62–78 |
| Gunicorn server workers | 17 |
| Dramatiq consumer threads (peak surge) | 36 |
| **Total at peak** | **~131** |

The channel layer consumes 78 of 85 available slots at the moment of exhaustion, confirmed by pg_stat_activity grouped output:

```
connections | state | query
-------------+---------+--------------------------------------------------
78 | idle | SELECT DISTINCT "django_channels_postgres_groupchannel"."channel" FROM ...
1 | active | (monitoring query)
1 | idle | INSERT INTO "authentik_tasks_tasklog" ...
1 | idle in transaction | SELECT ... accesstoken ...
```

**Workaround:** Raising `default_pool_size` to 200 stabilizes the system. At that level — 78 channels baseline + 17 gunicorn + 36 Dramatiq + ~20 buffer ≈ 151 — the pool has sufficient headroom. This is 40% of `max_connections = 500` on PostgreSQL, which is higher than typical recommendations for a single application.

**Prior behavior:** Prior to 2026.2.x, Authentik used a Redis-backed channel layer. `AUTHENTIK_REDIS__HOST` is configured and the Redis container is healthy in this deployment; the switch to `PostgresChannelLayer` as the default does not appear reflected in upgrade notes.

**Secondary issue (related but separate):** Authentik's task scheduler trigger `pgtrigger_notify_enqueueing_0bc94` fires a PostgreSQL NOTIFY on every task enqueue with `state='queued'`. When multiple scheduled tasks complete simultaneously (a cron coincidence), the resulting NOTIFY burst wakes all 36 Dramatiq consumer threads at once. Each thread calls `_fetch_pending_messages` using `SELECT FOR UPDATE`, blocking on row-level locks and holding a pool slot for the duration. This is a distinct pool exhaustion path from the channel layer issue but is made catastrophic by the reduced headroom.

**Questions for maintainers:**
1. Is `PostgresChannelLayer` intended for deployments using PgBouncer in transaction pooling mode, or primarily for direct-to-PostgreSQL setups?
2. Is there a supported configuration to cap the number of concurrent channel layer connections?
3. Is the Redis channel layer still supported via configuration in 2026.2.x?
4. Are minimum `pool_size` requirements planned for documentation?

### Deployment Method

Docker

### Version

2026.2.2

### Relevant log output

```shell
**Worker log at pool exhaustion (stack trace):**

{"event": "Consumer encountered an unexpected error.",
"exception": [{"exc_type": "ProtocolViolation", "exc_value": "query_wait_timeout",
"frames": [
{"filename": "dramatiq/worker.py", "lineno": 265, "name": "run"},
{"filename": "django_dramatiq_postgres/broker.py", "lineno": 62, "name": "wrapper"},
{"filename": "django_dramatiq_postgres/broker.py", "lineno": 409, "name": "__next__"},
{"filename": "django_dramatiq_postgres/broker.py", "lineno": 300, "name": "_fetch_pending_messages"},
{"filename": "django/db/models/query.py", "lineno": 386, "name": "__iter__"},
{"filename": "django/db/backends/base/base.py", "lineno": 320, "name": "cursor"},
{"filename": "psycopg/cursor.py", "lineno": 117, "name": "execute"}
]}],
"level": "critical",
"logger": "dramatiq.worker.ConsumerThread(default)",
"timestamp": "2026-05-17T05:32:58.284628"}


**Server log during cascade:**

{"error": "context canceled", "event": "failed to proxy to backend",
"level": "warning", "logger": "authentik.router", "timestamp": "2026-05-17T20:32:03Z"}
{"error": "context canceled", "event": "failed to proxy to backend",
"level": "warning", "logger": "authentik.router", "timestamp": "2026-05-17T20:32:11Z"}


**LDAP outpost during cascade:**

{"took-ms": 242303, "event": "Bind request", "level": "info", "timestamp": "..."}
{"took-ms": 254526, "event": "Bind request", "level": "info", "timestamp": "..."}
{"took-ms": 257042, "event": "Bind request", "level": "info", "timestamp": "..."}

*(Bind requests taking 4+ minutes — backends unavailable due to pool exhaustion)*

**PgBouncer pool state at onset (from pg_stat_activity):**

SELECT count(*) AS channels_conns
FROM pg_stat_activity
WHERE query LIKE '%django_channels_postgres_groupchannel%';

channels_conns
----------------
62 ← immediately post-boot, no user traffic
(1 row)
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the PostgresChannelLayer connection usage and the django_channels_postgres_groupchannel queries described in the report. Then inspect django_dramatiq_postgres/broker.py, especially _fetch_pending_messages, alongside the pgtrigger_notify_enqueueing_0bc94 behavior. Reproduce the reported baseline and burst under PgBouncer, and establish a supported configuration or documented behavior that prevents the reported pool exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, postgresql, python
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.