elastic / elastic/fleet-server

Ghost agent documents created when ES is disrupted during enrollment

Open
#7,741 6 comments 0 reactions 1 assignee Claimed by @ycombinator View on GitHub
bug Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
113
Forks
117
Avg merge
1d 16h
Merged PRs (30d)
112

Description

## Summary

Under ES load, fleet-server can leave orphaned `.fleet-agents` documents — agents that enrolled but never checked in. These "ghost" agents inflate the `active` count in the Fleet agent status API above the true `online` count.

## What is a ghost agent

A ghost agent is a `.fleet-agents` document that:
- Has an `enrolled_at` timestamp (fleet-server wrote it successfully)
- Has **no `last_checkin` field** (the agent never contacted Fleet after enrollment)
- Is counted as `active` but not `online` in the Fleet agent status API

## Evidence

Observed in consecutive daily 30k checkin scale test runs:

| Build | Ghosts (`all - online`) | Notes |
|-------|------------------------|-------|
| [#6895](https://buildkite.com/elastic/observability-perf/builds/6895) | 232 | ES autoscale event; bulk write timeouts during enrollment window 00:26–00:37 UTC |
| [#6899](https://buildkite.com/elastic/observability-perf/builds/6899) | 79 | Shorter disruption |
| [#6901](https://buildkite.com/elastic/observability-perf/builds/6901) | 0 | No ES disruption; clean run |

Ghost `enrolled_at` timestamps span the ES disruption window exactly. Total agent documents created exceeded the requested count by the ghost count (e.g. 30,232 documents for 30,000 requested agents), indicating re-enrollments occurred.

## Related

- observability-perf test workaround: elastic/observability-perf#937

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.