goauthentik / goauthentik/authentik
Paginated LDAP sync floods System Tasks, Redis, and the worker queue
- Dominant language
- Python
- Stars
- 25.6k
- Forks
- 2k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 644
Description
## What
Same LDAP sync-at-scale problem (same reporter): paginated ldap_sync floods System Tasks / Redis / the worker queue and does not clean up on failure. #6876 explicitly cites #6811 and #6816.
## Want
- Does not clean up on failure
- #6876 explicitly cites #6811
## Ideas this came from
- [#6811 ldap_sync_paginator: Error handling](https://github.com/goauthentik/authentik/issues/6811) — When ldapsyncpaginator has to fetch a large number of pages (25k in my case), it will create a lot of redis entries. If the method fails at any point, these ent
- [#6816 ldap_sync for many users is spamming the System Tasks](https://github.com/goauthentik/authentik/issues/6816) — Due to pagination, one instance of the ldapsync task is created for each 50 users from an LDAP source. My LDAP source has more than 100,000 users. This in itsel
- [#6844 ldap_sync is blocking the authentik task queue](https://github.com/goauthentik/authentik/issues/6844) — ldapsyncall has created about 6000 ldapsync tasks on our system. Currently, we are able to process about 8 of them per minute, meaning they will be running for
- [#6876 Improvement to ldap_sync task structure](https://github.com/goauthentik/authentik/issues/6876) — The way that the ldap sync tasks are currently structured creates a few problems (e.g., #6811 #6816). Also, timeouts aren't handled properly (in my opinion, the
Contributor guide
Research direction
Start by reading the linked issues #6811, #6816, #6844, and #6876, then trace the paginated ldap_sync, ldapsyncpaginator, and ldapsyncall entry points. Done means large LDAP syncs no longer flood System Tasks, Redis, or the worker queue, and failures clean up their work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- authentication, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100