goauthentik / goauthentik/authentik

PolicyEngine deadlock when User.attributes contains large data (e.g. avatar)

Open
#20,120 3 comments 0 reactions 0 assignees View on GitHub
bug bug/confirmed
Dominant language
Python
Stars
25.6k
Forks
2k
Avg merge
1d 1h
Merged PRs (30d)
644

Description

### Describe the bug

## Summary

The `PolicyEngine` deadlocks when evaluating policy bindings that involve a `User` whose `attributes` field contains large data (e.g., a base64-encoded avatar image). The `multiprocessing.Pipe` used for IPC
between the policy evaluation subprocess and the parent process has a fixed OS buffer size (64KB on Linux). When the serialized `PolicyResult` exceeds this buffer, `Pipe.send()` blocks indefinitely because the
reader hasn't started consuming yet — creating an unrecoverable deadlock.

This causes the `/application/o/authorize/` endpoint to hang forever for authenticated users, completely breaking OIDC login flows.

### How to reproduce

## Steps to Reproduce

1. Upload an avatar image for any user (e.g., via the Authentik admin UI or API). A normal JPEG photo (~2MB) is sufficient.
2. Create an Application with at least one policy binding — either a **user binding** referencing that user, or a **group binding** where that user is a member (since the `PolicyRequest` also contains the
user).
3. Authenticate as that user and access the `/application/o/authorize/` endpoint (i.e., perform a normal OIDC login flow).
4. The request hangs indefinitely. No log entry is produced. No error is returned.

### Expected behavior

## Expected Behavior

The authorize endpoint should process the request and return a redirect (302) to the authorization flow, regardless of the size of the user's `attributes` field.

## Actual Behavior

The authorize endpoint hangs forever. The gunicorn worker thread is blocked on `Pipe.send()` and never recovers. The request produces no ASGI access log entry (since logging happens after the response).

## Root Cause Analysis

The deadlock occurs in `PolicyEngine.build()` (`authentik/policies/engine.py`):

```python
# Phase 1: Evaluate all bindings sequentially
for binding in self.iterate_bindings():
our_end, task_end = Pipe(False)
task = PolicyProcess(binding, self.request, task_end)
task.daemon = False
if not CURRENT_PROCESS._config.get("daemon"):
task.run() # <-- synchronous: calls connection.send(result)
else:
task.start() # <-- async: spawns subprocess
self.__processes.append(PolicyProcessInfo(process=task, connection=our_end, binding=binding))

# Phase 2: Read results (only reached AFTER all bindings are processed)
for proc_info in self.__processes:
if proc_info.process.is_alive():
proc_info.process.join(proc_info.binding.timeout)
if not proc_info.result:
proc_info.result = proc_info.connection.recv()

The issue chain:

1. PolicyProcess.execute() sets policy_result.source_binding = self.binding on the result
2. PolicyProcess.run() calls self.connection.send(result) which pickles the PolicyResult
3. The pickle includes source_binding → PolicyBinding → User → User.attributes (containing the avatar)
4. pickle.dumps() on a User with a 2MB avatar produces a ~2.7MB payload
5. multiprocessing.Pipe has a 64KB OS buffer on Linux
6. send() writes 64KB, then blocks waiting for the reader to drain the buffer
7. The reader (recv()) is in Phase 2, which only runs after Phase 1 completes
8. Phase 1 is blocked on send() → deadlock

Verification

# In ak shell:
from authentik.core.models import User
import pickle

user = User.objects.get(username='django_admin')
print(len(pickle.dumps(user))) # 2,709,191 bytes (has avatar)
print(len(pickle.dumps(user.attributes))) # 2,708,497 bytes (the avatar)

# The pipe buffer is 64KB — this will never fit

Environment

- authentik version: 2025.6.4
- Platform: Linux (Docker, kernel 5.15)
- Python: 3.12
- Trigger: Uploading a standard JPEG avatar (~2MB) for a user referenced in application policy bindings

Impact

- Severity: High — completely breaks OIDC login for any application with policy bindings when any referenced user has a non-trivial avatar
- Affected endpoints: /application/o/authorize/ (and potentially any endpoint that triggers PolicyEngine evaluation)
- User-facing: Browser shows infinite loading spinner on the Authentik authorization page after entering credentials
- Recovery: Only workaround is removing all policy bindings from the application or removing the avatar from the user

Suggested Fixes

Several approaches could resolve this:

1. Don't include source_binding in the serialized result — source_binding is set in PolicyProcess.execute() for debugging/logging, but it forces the entire binding (and its related User/Group) through the
pipe. The binding reference could be passed separately or looked up by ID after recv().
2. Exclude heavy fields from User pickling — Implement __getstate__/__reduce__ on the User model to exclude attributes (or at least large blob values) during pickle serialization.
3. Use a different IPC mechanism — Replace multiprocessing.Pipe with a mechanism that doesn't have fixed buffer limits (e.g., multiprocessing.Queue, shared memory, or temporary files for large payloads).
4. Read results eagerly — Instead of the two-phase approach (write all, then read all), read each result immediately after the corresponding task completes to prevent buffer accumulation.
5. Limit avatar storage size — Enforce a maximum size for data stored in User.attributes, or store avatars as file references rather than inline base64 blobs.

### Screenshots

_No response_

### Additional context

_No response_

### Deployment Method

Docker

### Version

2025.6.4

### Relevant log output

```shell

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.