elastic / elastic/fleet-server
monitor: checkpoint must not advance when search returns nil due to unavailable index or shard
- Dominant language
- Go
- Stars
- 113
- Forks
- 117
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 112
Description
## Summary
On a fresh serverless (stateless ES) deployment, the first action written to `.fleet-actions` is silently dropped when a second action is submitted before the catch-up path rescues it.
## Root cause
`monitor.go:search()` swallows `index_not_found_exception` and returns `nil, nil` — treating a failed search identically to a genuinely empty result. Zero hits falls through to `m.storeCheckpoint(newCheckpoint)`, advancing the checkpoint past `_seq_no: 0` even though nothing was dispatched.
**Why it happens on stateless ES:** When the first action is written it auto-creates `.fleet-actions-7`. The stateless ES search shard for a newly created index may not be `STARTED` yet — `FleetSearch` with `wait_for_checkpoints=[0]` fails. The monitor treats this as "no documents", advances the checkpoint, and the action is never dispatched.
**Why catch-up also fails:** `FindAgentActions(seqno, checkpoint)` queries `_seq_no > seqno`. If a second action (`_seq_no: 1`) is submitted and ACKd before the catch-up runs, the agent's `seqno` becomes 1. All subsequent catch-up queries use `_seq_no > 1`, permanently excluding the action at `_seq_no: 0`.
## Evidence
Reproduced on two fresh serverless projects.
**Endpoint response action (`running-processes`):**
- Action `c06b42b0` → `_seq_no: 0`, `nbAgentsAck: 0`, permanently stuck
- Action `d401b044` → `_seq_no: 1`, `nbAgentsAck: 1`, delivered in ~2s
- Only 2 documents in `.fleet-actions-7`
- Endpoint log confirms `c06b42b0` never received; `d401b044` received within 2s of creation
**`REQUEST_DIAGNOSTICS` (no Endpoint):**
- Action `773d3d6f` → `_seq_no: 0`, `nbAgentsAck: 0`, permanently stuck
- Action `43b26e52` → `_seq_no: 1`, `nbAgentsAck: 1`, delivered in 7s
- Only 2 documents in `.fleet-actions-7`
Single-action test: a `REQUEST_DIAGNOSTICS` at `_seq_no: 0` with no retry was rescued by catch-up after ~3.5 minutes — confirming the permanent hang requires a second action to be ACKd first.
Related ES bug: elastic/elasticsearch#130555 ("Fleet search using `wait_for_checkpoints` can fail if the node executing the search is recovering").
## Fix
`monitor.go:search()` must distinguish between a search that genuinely returned zero hits and one that failed because the index or search shard was unavailable. The checkpoint should only advance when the search genuinely completed — not when it returned `nil, nil` due to `index_not_found` or shard unavailability.
## Kibana issue
elastic/kibana#281980
Contributor guide
Research direction
Start at monitor.go:search() and trace how its nil, nil result is handled before storeCheckpoint(newCheckpoint). Reproduce the two-action sequence described in the issue, then verify that genuinely empty searches still advance the checkpoint while index or shard-unavailable searches do not and the first action is later dispatched.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100