goauthentik / goauthentik/authentik
PolicyEngine deadlock when User.attributes contains large data (e.g. avatar)
- Dominant language
- Python
- Stars
- 25.6k
- Forks
- 2k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 644
Description
### Describe the bug
## Summary
The `PolicyEngine` deadlocks when evaluating policy bindings that involve a `User` whose `attributes` field contains large data (e.g., a base64-encoded avatar image). The `multiprocessing.Pipe` used for IPC
between the policy evaluation subprocess and the parent process has a fixed OS buffer size (64KB on Linux). When the serialized `PolicyResult` exceeds this buffer, `Pipe.send()` blocks indefinitely because the
reader hasn't started consuming yet — creating an unrecoverable deadlock.
This causes the `/application/o/authorize/` endpoint to hang forever for authenticated users, completely breaking OIDC login flows.
### How to reproduce
## Steps to Reproduce
1. Upload an avatar image for any user (e.g., via the Authentik admin UI or API). A normal JPEG photo (~2MB) is sufficient.
2. Create an Application with at least one policy binding — either a **user binding** referencing that user, or a **group binding** where that user is a member (since the `PolicyRequest` also contains the
user).
3. Authenticate as that user and access the `/application/o/authorize/` endpoint (i.e., perform a normal OIDC login flow).
4. The request hangs indefinitely. No log entry is produced. No error is returned.
### Expected behavior
## Expected Behavior
The authorize endpoint should process the request and return a redirect (302) to the authorization flow, regardless of the size of the user's `attributes` field.
## Actual Behavior
The authorize endpoint hangs forever. The gunicorn worker thread is blocked on `Pipe.send()` and never recovers. The request produces no ASGI access log entry (since logging happens after the response).
## Root Cause Analysis
The deadlock occurs in `PolicyEngine.build()` (`authentik/policies/engine.py`):
```python
# Phase 1: Evaluate all bindings sequentially
for binding in self.iterate_bindings():
our_end, task_end = Pipe(False)
task = PolicyProcess(binding, self.request, task_end)
task.daemon = False
if not CURRENT_PROCESS._config.get("daemon"):
task.run() # <-- synchronous: calls connection.send(result)
else:
task.start() # <-- async: spawns subprocess
self.__processes.append(PolicyProcessInfo(process=task, connection=our_end, binding=binding))
# Phase 2: Read results (only reached AFTER all bindings are processed)
for proc_info in self.__processes:
if proc_info.process.is_alive():
proc_info.process.join(proc_info.binding.timeout)
if not proc_info.result:
proc_info.result = proc_info.connection.recv()
The issue chain:
1. PolicyProcess.execute() sets policy_result.source_binding = self.binding on the result
2. PolicyProcess.run() calls self.connection.send(result) which pickles the PolicyResult
3. The pickle includes source_binding → PolicyBinding → User → User.attributes (containing the avatar)
4. pickle.dumps() on a User with a 2MB avatar produces a ~2.7MB payload
5. multiprocessing.Pipe has a 64KB OS buffer on Linux
6. send() writes 64KB, then blocks waiting for the reader to drain the buffer
7. The reader (recv()) is in Phase 2, which only runs after Phase 1 completes
8. Phase 1 is blocked on send() → deadlock
Verification
# In ak shell:
from authentik.core.models import User
import pickle
user = User.objects.get(username='django_admin')
print(len(pickle.dumps(user))) # 2,709,191 bytes (has avatar)
print(len(pickle.dumps(user.attributes))) # 2,708,497 bytes (the avatar)
# The pipe buffer is 64KB — this will never fit
Environment
- authentik version: 2025.6.4
- Platform: Linux (Docker, kernel 5.15)
- Python: 3.12
- Trigger: Uploading a standard JPEG avatar (~2MB) for a user referenced in application policy bindings
Impact
- Severity: High — completely breaks OIDC login for any application with policy bindings when any referenced user has a non-trivial avatar
- Affected endpoints: /application/o/authorize/ (and potentially any endpoint that triggers PolicyEngine evaluation)
- User-facing: Browser shows infinite loading spinner on the Authentik authorization page after entering credentials
- Recovery: Only workaround is removing all policy bindings from the application or removing the avatar from the user
Suggested Fixes
Several approaches could resolve this:
1. Don't include source_binding in the serialized result — source_binding is set in PolicyProcess.execute() for debugging/logging, but it forces the entire binding (and its related User/Group) through the
pipe. The binding reference could be passed separately or looked up by ID after recv().
2. Exclude heavy fields from User pickling — Implement __getstate__/__reduce__ on the User model to exclude attributes (or at least large blob values) during pickle serialization.
3. Use a different IPC mechanism — Replace multiprocessing.Pipe with a mechanism that doesn't have fixed buffer limits (e.g., multiprocessing.Queue, shared memory, or temporary files for large payloads).
4. Read results eagerly — Instead of the two-phase approach (write all, then read all), read each result immediately after the corresponding task completes to prevent buffer accumulation.
5. Limit avatar storage size — Enforce a maximum size for data stored in User.attributes, or store avatars as file references rather than inline base64 blobs.
### Screenshots
_No response_
### Additional context
_No response_
### Deployment Method
Docker
### Version
2025.6.4
### Relevant log output
```shell
```
Contributor guide
Assessment
This issue has not been assessed yet.