goauthentik / goauthentik/authentik

Google Workspace and Microsoft Entra dispatch tasks flood queue during blueprint reconciliation

Open
#22,905 2 comments 0 reactions 0 assignees View on GitHub
bug bug/confirmed
Dominant language
Python
Stars
25.6k
Forks
2k
Avg merge
1d 2h
Merged PRs (30d)
659

Description

## Summary

Google Workspace and Microsoft Entra outgoing sync can turn normal blueprint reconciliation into a very large background-task backlog.

The core issue is that every saved `User` or `Group` emits a provider-type direct-dispatch task as soon as at least one provider of that type exists. The signal does this before authentik checks whether the saved object is selected by, assigned to, or in scope for any concrete provider.

In a declarative/GitOps migration, blueprint reconciliation can save many existing groups repeatedly. Once Google Workspace and Microsoft Entra providers exist, those saves can produce thousands of fresh dispatch tasks, even for groups that are not in either provider's scope.

## Why this matters

The task volume scales with saved objects, provider types, and reconciliation passes:

```text
saved users/groups
* outgoing provider signal families present
* repeated blueprint apply/discovery/startup/file-watch passes
= direct-dispatch task rows
```

For example, if a migration saves roughly 100 groups during one reconciliation pass and both enterprise provider types are present, that pass can enqueue roughly 200 provider-type dispatch tasks. Repeated apply/discovery/startup/file-watch events during an active migration can multiply that into thousands of pending tasks.

Dry-run providers do not prevent this. Dry-run affects mutating provider writes later in the sync path, but it does not suppress `User`/`Group` save signals or the initial direct-dispatch task creation.

## What produced the repeated waves in the observed deployment

The repeated blueprint waves were not caused by failed Google Workspace or Microsoft Entra sync tasks recursively triggering more blueprint applies. The failures made the generated sync work slow or sticky, but they were downstream of task creation.

In the observed Kubernetes deployment, the producers were Flux, the mounted blueprint ConfigMap, worker startup discovery, and per-worker file watchers:

- Blueprints are generated into one stable-name ConfigMap, `authentik-prod-blueprints`, from 31 blueprint files in `apps/authentik-prod/blueprints/kustomization.yaml`.
- The authentik HelmRelease mounts that ConfigMap into the worker pods through `blueprints.configMaps`.
- There are two authentik worker pods, and each worker starts a blueprint filesystem watcher for the mounted blueprint directory.
- Blueprint discovery is also registered with `send_on_startup=True`, so worker restarts enqueue discovery on startup.
- `app-authentik-prod-blueprints` is a Flux Kustomization with a 10 minute interval. Its status history showed multiple applied Git revisions during the migration window, including revisions first reconciled around 2026-06-07 18:55, 19:27, 19:44, 19:59, and 20:41 UTC.
- The authentik HelmRelease also rolled out at about 2026-06-07 19:59 UTC, recreating the server and worker pods. That creates another startup-discovery source while the ConfigMap changes are also being mounted.

A single ConfigMap update can therefore fan out into several authentik-level events:

```text
Flux applies updated ConfigMap
-> Kubernetes updates projected files in each worker pod
-> each worker's blueprint watcher sees file create/modify events
-> file creation events enqueue blueprints_discovery
-> file modification events enqueue apply_blueprint for matching instances
-> worker restarts also enqueue startup blueprints_discovery
```

There is no debounce/coalescing at these boundaries. If several discoveries run while applies are still queued or running, each discovery can still see `last_applied_hash != file_hash` and enqueue another `apply_blueprint` for the same blueprint instance. `apply_blueprint` sets a task UID only after the task starts; the initial enqueue is still a fresh Dramatiq message, so duplicate applies can already be in the queue.

That explains why the observed deployment had many apparent apply waves without needing a failed sync task to trigger the next wave. The sync failures matter because they kept the generated Google Workspace work from draining, but the repeated producer was the blueprint ConfigMap/update/startup/watch path.

## Observed behavior

During an active authentik migration with blueprint-managed applications, groups, and providers, the task queue filled with large numbers of fresh Google Workspace and Microsoft Entra direct-dispatch tasks, including:

```text
authentik.enterprise.providers.google_workspace.tasks.google_workspace_sync_direct_dispatch
authentik.enterprise.providers.microsoft_entra.tasks.microsoft_entra_sync_direct_dispatch
```

The task arguments were primarily group saves:

```text
('authentik.core.models.Group', '')
```

Many of these rows had retry count 0, which is important: the queue growth was not primarily caused by failing tasks retrying. It was caused by new dispatch tasks being created faster than they could drain.

## Relevant code path

### 1. Enterprise outgoing providers register broad save signals

Google Workspace and Microsoft Entra both register `User` and `Group` `post_save` handlers through `register_signals()`:

- `authentik/enterprise/providers/google_workspace/signals.py`
- `authentik/enterprise/providers/microsoft_entra/signals.py`
- `authentik/lib/sync/outgoing/signals.py`

The shared save handler checks only whether any provider of that type exists:

```python
if _CTX_INHIBIT_DISPATCH.get():
return
if not provider_type.objects.exists():
return
task_sync_direct_dispatch.send(
class_to_path(instance.__class__),
instance.pk,
)
```

This means every saved group emits one Google Workspace dispatch task if any Google Workspace provider exists, and one Microsoft Entra dispatch task if any Microsoft Entra provider exists.

### 2. Provider scope is checked only after the dispatch task already exists

`authentik/lib/sync/outgoing/tasks.py::sync_signal_direct_dispatch()` fans out to assigned providers:

```python
for provider in self._provider_model.objects.filter(
Q(backchannel_application__isnull=False) | Q(application__isnull=False)
):
task_sync_signal_direct.send_with_options(
args=(model, pk, provider.pk),
rel_obj=provider,
uid=f"{provider.name}:{model_class._meta.model_name}:{pk}:direct",
)
```

The object-level provider scope check happens later in `sync_signal_direct()`:

```python
queryset = provider.get_object_qs(instance.__class__, pk=instance.pk)
if not queryset or not queryset.exists():
return
```

So an out-of-scope group still costs at least one provider-type dispatch task per enterprise provider family before authentik discovers that no provider should sync it.

### 3. Blueprint apply can save existing groups

`authentik/blueprints/v1/importer.py::_apply_models()` calls `serializer.save()` for `PRESENT` entries. Re-applying a blueprint can therefore save existing group rows even when the effective configuration did not change.

That makes blueprint reconciliation an effective producer for these broad outgoing-sync `post_save` side effects.

### 4. Blueprint reconciliation can run repeatedly during migrations

During normal operations, blueprints can be applied by discovery, startup schedule reconciliation, file-watch events, and manual/API-triggered applies:

- `authentik/blueprints/v1/tasks.py::blueprints_discovery()` scans mounted blueprint files.
- `check_blueprint_v1_file()` enqueues `apply_blueprint` when file hashes change.
- `BlueprintWatcher` enqueues `apply_blueprint` on file modification events.
- `authentik/blueprints/apps.py` registers discovery with `send_on_startup=True`.

During active migrations and pod restarts, this gives authentik multiple opportunities to re-save blueprint-managed groups.

### 5. The first dispatch task is not coalesced by object

The provider-specific direct task uses a stable `uid`, but that happens after the higher-level provider-type dispatch task already exists.

`task_sync_direct_dispatch.send(...)` creates a fresh message each time, so repeated saves of the same group can create many separate `google_workspace_sync_direct_dispatch` and `microsoft_entra_sync_direct_dispatch` rows.

## Expected behavior

authentik should keep declarative reconciliation from flooding the task system with enterprise outgoing-sync dispatch work. In particular:

- Re-applying unchanged blueprint-managed groups should not enqueue large numbers of outgoing direct-sync tasks.
- Google Workspace and Microsoft Entra dispatch tasks should be coalesced or deduplicated by provider type, model, and primary key while queued/running.
- Provider scope should be checked before, or much closer to, task creation when practical.
- Dry-run providers should not unexpectedly multiply background task volume unless that is intentional and documented simulation behavior.
- Blueprint validation/rollback paths should not leak outgoing sync side effects.

## Suggested reproduction

1. Start authentik with the PostgreSQL Dramatiq broker enabled.
2. Configure Google Workspace and Microsoft Entra outgoing providers, optionally in dry-run mode.
3. Mount or apply blueprints that manage many `authentik_core.group` entries using normal `PRESENT` semantics.
4. Trigger repeated blueprint reconciliation, for example through startup discovery, worker/server restarts, mounted-file changes, or repeated manual/API blueprint applies.
5. Inspect task rows for these actors:

```text
google_workspace_sync_direct_dispatch
microsoft_entra_sync_direct_dispatch
```

Useful SQL shape:

```sql
select actor_name, state, count(*), min(mtime), max(mtime)
from
where actor_name in (
'authentik.enterprise.providers.google_workspace.tasks.google_workspace_sync_direct_dispatch',
'authentik.enterprise.providers.microsoft_entra.tasks.microsoft_entra_sync_direct_dispatch'
)
group by actor_name, state
order by actor_name, state;
```

To confirm the producer side, instrument:

- `authentik/lib/sync/outgoing/signals.py::model_post_save`
- `authentik/blueprints/v1/importer.py::_apply_models`
- `authentik/lib/sync/outgoing/tasks.py::sync_signal_direct_dispatch`

## Possible fixes to evaluate

These are not mutually exclusive:

1. Avoid saving unchanged existing blueprint-managed objects.
- If validated blueprint data matches the persisted object, skip `serializer.save()`.
- This addresses no-op blueprint apply storms at the source.

2. Coalesce direct-dispatch tasks by `(provider type, model, pk)`.
- The provider-specific direct task already gets a stable `uid`.
- The higher-level dispatch actor should get similar coalescing so repeated saves do not create thousands of identical pending rows.

3. Move scope filtering earlier.
- If no provider of that type can sync the object, skip enqueueing follow-up work.
- This may be more expensive than dedupe, but it avoids pointless fanout for out-of-scope groups.

4. Suppress outgoing dispatch during blueprint validation/rollback.
- Validation runs in a rollback context and should not emit persistent task side effects.
- `sync_outgoing_inhibit_dispatch()` and/or `transaction.on_commit()` should be evaluated here.

5. Define dry-run scheduling semantics.
- If dry-run is intended to simulate all scheduling, document that it can still enqueue direct-sync tasks.
- If dry-run is intended to be operationally quiet, skip signal-driven direct sync for dry-run providers or make it configurable.

## Impact

This can saturate the task system during normal declarative operations. Environments that manage groups, applications, and providers through blueprints can see a sudden backlog when adding Google Workspace or Microsoft Entra outgoing providers, especially during migrations and restarts.

The practical result is that adding enterprise outgoing providers can make authentik appear stuck or noisy even when most affected groups are not actually in scope for those providers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.