elastic / elastic/fleet-server

monitor: checkpoint must not advance when search returns nil due to unavailable index or shard

Open
#7,558 2 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
113
Forks
117
Avg merge
1d 16h
Merged PRs (30d)
112

Description

## Summary

On a fresh serverless (stateless ES) deployment, the first action written to `.fleet-actions` is silently dropped when a second action is submitted before the catch-up path rescues it.

## Root cause

`monitor.go:search()` swallows `index_not_found_exception` and returns `nil, nil` — treating a failed search identically to a genuinely empty result. Zero hits falls through to `m.storeCheckpoint(newCheckpoint)`, advancing the checkpoint past `_seq_no: 0` even though nothing was dispatched.

**Why it happens on stateless ES:** When the first action is written it auto-creates `.fleet-actions-7`. The stateless ES search shard for a newly created index may not be `STARTED` yet — `FleetSearch` with `wait_for_checkpoints=[0]` fails. The monitor treats this as "no documents", advances the checkpoint, and the action is never dispatched.

**Why catch-up also fails:** `FindAgentActions(seqno, checkpoint)` queries `_seq_no > seqno`. If a second action (`_seq_no: 1`) is submitted and ACKd before the catch-up runs, the agent's `seqno` becomes 1. All subsequent catch-up queries use `_seq_no > 1`, permanently excluding the action at `_seq_no: 0`.

## Evidence

Reproduced on two fresh serverless projects.

**Endpoint response action (`running-processes`):**
- Action `c06b42b0` → `_seq_no: 0`, `nbAgentsAck: 0`, permanently stuck
- Action `d401b044` → `_seq_no: 1`, `nbAgentsAck: 1`, delivered in ~2s
- Only 2 documents in `.fleet-actions-7`
- Endpoint log confirms `c06b42b0` never received; `d401b044` received within 2s of creation

**`REQUEST_DIAGNOSTICS` (no Endpoint):**
- Action `773d3d6f` → `_seq_no: 0`, `nbAgentsAck: 0`, permanently stuck
- Action `43b26e52` → `_seq_no: 1`, `nbAgentsAck: 1`, delivered in 7s
- Only 2 documents in `.fleet-actions-7`

Single-action test: a `REQUEST_DIAGNOSTICS` at `_seq_no: 0` with no retry was rescued by catch-up after ~3.5 minutes — confirming the permanent hang requires a second action to be ACKd first.

Related ES bug: elastic/elasticsearch#130555 ("Fleet search using `wait_for_checkpoints` can fail if the node executing the search is recovering").

## Fix

`monitor.go:search()` must distinguish between a search that genuinely returned zero hits and one that failed because the index or search shard was unavailable. The checkpoint should only advance when the search genuinely completed — not when it returned `nil, nil` due to `index_not_found` or shard unavailability.

## Kibana issue

elastic/kibana#281980

Contributor guide

Open the contributing guide

Research direction

Start at monitor.go:search() and trace how its nil, nil result is handled before storeCheckpoint(newCheckpoint). Reproduce the two-action sequence described in the issue, then verify that genuinely empty searches still advance the checkpoint while index or shard-unavailable searches do not and the first action is later dispatched.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.