elastic / elastic/fleet-server
Ghost agent documents created when ES is disrupted during enrollment
- Dominant language
- Go
- Stars
- 113
- Forks
- 117
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 112
Description
## Summary
Under ES load, fleet-server can leave orphaned `.fleet-agents` documents — agents that enrolled but never checked in. These "ghost" agents inflate the `active` count in the Fleet agent status API above the true `online` count.
## What is a ghost agent
A ghost agent is a `.fleet-agents` document that:
- Has an `enrolled_at` timestamp (fleet-server wrote it successfully)
- Has **no `last_checkin` field** (the agent never contacted Fleet after enrollment)
- Is counted as `active` but not `online` in the Fleet agent status API
## Evidence
Observed in consecutive daily 30k checkin scale test runs:
| Build | Ghosts (`all - online`) | Notes |
|-------|------------------------|-------|
| [#6895](https://buildkite.com/elastic/observability-perf/builds/6895) | 232 | ES autoscale event; bulk write timeouts during enrollment window 00:26–00:37 UTC |
| [#6899](https://buildkite.com/elastic/observability-perf/builds/6899) | 79 | Shorter disruption |
| [#6901](https://buildkite.com/elastic/observability-perf/builds/6901) | 0 | No ES disruption; clean run |
Ghost `enrolled_at` timestamps span the ES disruption window exactly. Total agent documents created exceeded the requested count by the ghost count (e.g. 30,232 documents for 30,000 requested agents), indicating re-enrollments occurred.
## Related
- observability-perf test workaround: elastic/observability-perf#937
Contributor guide
Assessment
This issue has not been assessed yet.