elastic / elastic/elastic-agent
Make Fleet-issued UNENROLL actions recoverable (use long-wait scheduler instead of permanently stopping the gateway)
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
## Summary
[#5428](https://github.com/elastic/elastic-agent/issues/5428) made **self-triggered** unenrollment (after 7 consecutive 401s) recoverable by switching to a long-wait scheduler (1 hour) instead of permanently stopping the Fleet gateway. This means agents in disaster recovery scenarios can automatically reconnect once their API key is restored — without manual intervention on each host.
However, **explicit UNENROLL actions issued by Fleet** still permanently stop the gateway. The handler unconditionally cancels the gateway context and persists the unenroll action to the state store:
```go
// handler_action_unenroll.go
// backup action for future start to avoid starting fleet gateway loop
backupCallback := func() {
h.stateStore.SetAction(a)
if err := h.stateStore.Save(); err != nil {
h.log.Warnf("Failed to update state store: %v", err)
}
}
stopComponents(ctx, h.ch, a, acker, backupCallback)
// close fleet gateway loop
for _, c := range h.closers {
c()
}
```
This means:
1. The fleet gateway emits `"Fleet gateway stopped"` and exits permanently
2. The unenroll action is persisted — so even a process restart keeps the agent dormant
3. The only recovery path is re-enrollment, which requires manual access to each host
## Problem
When a mass accidental unenrollment occurs (e.g. due to a Fleet bug that causes a bulk action to affect unintended agents), the agents cannot self-recover even after the Fleet-side state is corrected. Every affected host requires manual intervention to clear the state store and restart, or a full re-enrollment.
This is the same disaster recovery problem that motivated #5428 for the self-unenroll path — but the explicit UNENROLL path has the same gap.
## Proposal
Apply the same long-wait scheduler approach from #5428 to Fleet-issued UNENROLL actions:
- Instead of cancelling the closers (which permanently stops the gateway), switch to the long-wait scheduler (1 hour checkin interval)
- On a successful checkin — meaning Fleet responds with an active enrolled state — reset the `unauthCounter`, return to the normal scheduler duration, and resume normal operation
- Continue to stop components (send empty policy) as today, since that is the correct response to an UNENROLL action
- The state store persisted action should either not be saved or be cleared on successful checkin recovery
This allows an agent that was accidentally unenrolled (or whose Fleet-side state has been corrected) to automatically recover without any manual intervention on the host.
## Intentional unenrollment case
For agents that are deliberately and permanently unenrolled, the behaviour change is minimal: the agent process continues running (dormant, no components active) and checks in once per hour instead of never. This is consistent with the current zombie-process behaviour documented in [beats#24568](https://github.com/elastic/beats/issues/24568), and avoids the service manager restart loop that was the original motivation for keeping the process alive post-unenroll.
## Context
- [#5428](https://github.com/elastic/elastic-agent/issues/5428) — the self-unenroll fix this issue mirrors
- [elastic/elastic-agent#6619](https://github.com/elastic/elastic-agent/pull/6619) — implementing PR for #5428
- `internal/pkg/agent/application/actions/handlers/handler_action_unenroll.go` — where the explicit UNENROLL is handled
- `internal/pkg/agent/application/gateway/fleet/fleet_gateway.go:36,56` — `maxUnauthCounter` and `ErrConsecutiveUnauthDuration` constants already in place
Contributor guide
Research direction
Start in internal/pkg/agent/application/actions/handlers/handler_action_unenroll.go, then compare the self-unenroll recovery from #5428 and PR #6619. Read the scheduler and state-store flow alongside internal/pkg/agent/application/gateway/fleet/fleet_gateway.go, including the existing unauthenticated-checkin constants. Done means Fleet-issued unenrollment waits hourly, recovers on an active checkin, and leaves intentional unenrollment dormant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100