goauthentik / goauthentik/authentik

Paginated LDAP sync floods System Tasks, Redis, and the worker queue

Open
#25,252 0 comments 0 reactions 0 assignees View on GitHub
bug/confirmed enhancement enhancement/confirmed
Dominant language
Python
Stars
25.6k
Forks
2k
Avg merge
1d 1h
Merged PRs (30d)
644

Description

## What

Same LDAP sync-at-scale problem (same reporter): paginated ldap_sync floods System Tasks / Redis / the worker queue and does not clean up on failure. #6876 explicitly cites #6811 and #6816.

## Want

- Does not clean up on failure
- #6876 explicitly cites #6811

## Ideas this came from

- [#6811 ldap_sync_paginator: Error handling](https://github.com/goauthentik/authentik/issues/6811) — When ldapsyncpaginator has to fetch a large number of pages (25k in my case), it will create a lot of redis entries. If the method fails at any point, these ent
- [#6816 ldap_sync for many users is spamming the System Tasks](https://github.com/goauthentik/authentik/issues/6816) — Due to pagination, one instance of the ldapsync task is created for each 50 users from an LDAP source. My LDAP source has more than 100,000 users. This in itsel
- [#6844 ldap_sync is blocking the authentik task queue](https://github.com/goauthentik/authentik/issues/6844) — ldapsyncall has created about 6000 ldapsync tasks on our system. Currently, we are able to process about 8 of them per minute, meaning they will be running for
- [#6876 Improvement to ldap_sync task structure](https://github.com/goauthentik/authentik/issues/6876) — The way that the ldap sync tasks are currently structured creates a few problems (e.g., #6811 #6816). Also, timeouts aren't handled properly (in my opinion, the

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked issues #6811, #6816, #6844, and #6876, then trace the paginated ldap_sync, ldapsyncpaginator, and ldapsyncall entry points. Done means large LDAP syncs no longer flood System Tasks, Redis, or the worker queue, and failures clean up their work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.