elastic / elastic/elastic-agent

Make Fleet-issued UNENROLL actions recoverable (use long-wait scheduler instead of permanently stopping the gateway)

Open
#13,621 5 comments 2 reactions 0 assignees View on GitHub
enhancement Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 23h
Merged PRs (30d)
312

Description

## Summary

[#5428](https://github.com/elastic/elastic-agent/issues/5428) made **self-triggered** unenrollment (after 7 consecutive 401s) recoverable by switching to a long-wait scheduler (1 hour) instead of permanently stopping the Fleet gateway. This means agents in disaster recovery scenarios can automatically reconnect once their API key is restored — without manual intervention on each host.

However, **explicit UNENROLL actions issued by Fleet** still permanently stop the gateway. The handler unconditionally cancels the gateway context and persists the unenroll action to the state store:

```go
// handler_action_unenroll.go
// backup action for future start to avoid starting fleet gateway loop
backupCallback := func() {
h.stateStore.SetAction(a)
if err := h.stateStore.Save(); err != nil {
h.log.Warnf("Failed to update state store: %v", err)
}
}

stopComponents(ctx, h.ch, a, acker, backupCallback)

// close fleet gateway loop
for _, c := range h.closers {
c()
}
```

This means:
1. The fleet gateway emits `"Fleet gateway stopped"` and exits permanently
2. The unenroll action is persisted — so even a process restart keeps the agent dormant
3. The only recovery path is re-enrollment, which requires manual access to each host

## Problem

When a mass accidental unenrollment occurs (e.g. due to a Fleet bug that causes a bulk action to affect unintended agents), the agents cannot self-recover even after the Fleet-side state is corrected. Every affected host requires manual intervention to clear the state store and restart, or a full re-enrollment.

This is the same disaster recovery problem that motivated #5428 for the self-unenroll path — but the explicit UNENROLL path has the same gap.

## Proposal

Apply the same long-wait scheduler approach from #5428 to Fleet-issued UNENROLL actions:

- Instead of cancelling the closers (which permanently stops the gateway), switch to the long-wait scheduler (1 hour checkin interval)
- On a successful checkin — meaning Fleet responds with an active enrolled state — reset the `unauthCounter`, return to the normal scheduler duration, and resume normal operation
- Continue to stop components (send empty policy) as today, since that is the correct response to an UNENROLL action
- The state store persisted action should either not be saved or be cleared on successful checkin recovery

This allows an agent that was accidentally unenrolled (or whose Fleet-side state has been corrected) to automatically recover without any manual intervention on the host.

## Intentional unenrollment case

For agents that are deliberately and permanently unenrolled, the behaviour change is minimal: the agent process continues running (dormant, no components active) and checks in once per hour instead of never. This is consistent with the current zombie-process behaviour documented in [beats#24568](https://github.com/elastic/beats/issues/24568), and avoids the service manager restart loop that was the original motivation for keeping the process alive post-unenroll.

## Context

- [#5428](https://github.com/elastic/elastic-agent/issues/5428) — the self-unenroll fix this issue mirrors
- [elastic/elastic-agent#6619](https://github.com/elastic/elastic-agent/pull/6619) — implementing PR for #5428
- `internal/pkg/agent/application/actions/handlers/handler_action_unenroll.go` — where the explicit UNENROLL is handled
- `internal/pkg/agent/application/gateway/fleet/fleet_gateway.go:36,56` — `maxUnauthCounter` and `ErrConsecutiveUnauthDuration` constants already in place

Contributor guide

Open the contributing guide

Research direction

Start in internal/pkg/agent/application/actions/handlers/handler_action_unenroll.go, then compare the self-unenroll recovery from #5428 and PR #6619. Read the scheduler and state-store flow alongside internal/pkg/agent/application/gateway/fleet/fleet_gateway.go, including the existing unauthenticated-checkin constants. Done means Fleet-issued unenrollment waits hourly, recovers on an active checkin, and leaves intentional unenrollment dormant.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.