elastic / elastic/elastic-agent
Repeated outbound probes when a policy change contains unreachable Fleet Server hosts
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 303
Description
### Problem
When a policy update changes the Fleet Server hosts, [`validateFleetServerHosts`](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L130-L157) calls [`testFleetConfig`](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L159), which sends a `GET /api/status` request (15 s timeout) to the candidate hosts. If the probe fails, `Handle()` returns an error and the action is not ACKed.
Fleet Server re-delivers every un-ACKed action on the next checkin (~1 s). Because `h.config.Fleet.Client` is only updated [after a successful probe](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L141), the [`clientEqual`](https://github.com/elastic/elastic-agent/blob/97f3e41da6c8f5ae52d0dfa0ff376cb9abe26b5b/internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go#L517) guard keeps returning false, so `testFleetConfig` fires again on every re-delivery — one 15 s probe per second, indefinitely.
### Impact
Overlapping HTTP connections accumulate as long as the new Fleet Server hosts are unreachable, generating unnecessary load on the agent and on whatever is at the target URL.
### Proposed fix
Track whether a probe is already in-flight on `PolicyChangeHandler`. If `testFleetConfig` is already running when `Handle()` is called again for the same action, skip the new probe and return the previous error immediately. The next `Handle()` after the in-flight probe completes would then either succeed (hosts reachable) or start a fresh probe.
This avoids accumulating concurrent connections with no cache TTL to tune, and self-heals naturally once the unreachable hosts become available.
Contributor guide
Research direction
Start in internal/pkg/agent/application/actions/handlers/handler_action_policy_change.go, reading Handle(), validateFleetServerHosts, testFleetConfig, and clientEqual. Trace repeated policy-change deliveries with unreachable Fleet Server hosts; done means an in-flight probe is not duplicated, its previous error is returned, and a later retry can start after completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100