elastic / elastic/elastic-agent

Repeated outbound probes when a policy change contains unreachable Fleet Server hosts

Open
#16,136 3 comments 1 reaction 0 assignees View on GitHub
bug Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 20h
Merged PRs (30d)
303

Description

### Problem

When a policy update changes the Fleet Server hosts, [`validateFleetServerHosts`](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L130-L157) calls [`testFleetConfig`](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L159), which sends a `GET /api/status` request (15 s timeout) to the candidate hosts. If the probe fails, `Handle()` returns an error and the action is not ACKed.

Fleet Server re-delivers every un-ACKed action on the next checkin (~1 s). Because `h.config.Fleet.Client` is only updated [after a successful probe](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L141), the [`clientEqual`](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L517) guard keeps returning false, so `testFleetConfig` fires again on every re-delivery — one 15 s probe per second, indefinitely.

### Impact

Overlapping HTTP connections accumulate as long as the new Fleet Server hosts are unreachable, generating unnecessary load on the agent and on whatever is at the target URL.

### Proposed fix

Track whether a probe is already in-flight on `PolicyChangeHandler`. If `testFleetConfig` is already running when `Handle()` is called again for the same action, skip the new probe and return the previous error immediately. The next `Handle()` after the in-flight probe completes would then either succeed (hosts reachable) or start a fresh probe.

This avoids accumulating concurrent connections with no cache TTL to tune, and self-heals naturally once the unreachable hosts become available.

Contributor guide

Open the contributing guide

Research direction

Start in internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go, reading Handle(), validateFleetServerHosts, testFleetConfig, and clientEqual. Trace repeated policy-change deliveries with unreachable Fleet Server hosts; done means an in-flight probe is not duplicated, its previous error is returned, and a later retry can start after completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.