elastic / elastic/elastic-agent
[Elastic Agent] Fleet managed Elastic Agent stays healthy when it can't connect to Fleet Server
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 303
Description
- Version: 7.16.2
- Operating System: CentOS 7 (but issue can probably happen with any Fleet managed Agent)
- Steps to Reproduce:
1. Install Fleet Managed Elastic Agent on a system. (.tar.gz install)
2. Wait for the Agent to register and become healthy
3. Interrupt connection between Elastic Agent and Fleet Server
4. Wait for Elastic Agent to start reporting connection error logs:
```json
{"log.level":"error","@timestamp":"2022-02-01T15:23:52.311Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://fleet-server.example.com:8220/api/fleet/agents//checkin?\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","ecs.version":"1.6.0"}
```
5. Check Elastic Agent Status, and see that it is healthy:
```bash
sudo /opt/Elastic/Agent/elastic-agent status
Status: HEALTHY
Message: (no message)
Applications:
* endpoint-security (HEALTHY)
Protecting with policy {2dc16c0f-ebc4-400a-bd5a-7afacc5e4370}
* filebeat (HEALTHY)
Running
* metricbeat (HEALTHY)
Running
* filebeat_monitoring (HEALTHY)
Running
* metricbeat_monitoring (HEALTHY)
Running
* osquerybeat (HEALTHY)
Running
```
### Expected:
I'd expect the top level status to be UNHEALTHY as the Agent can no longer talk to Fleet server and therefore no longer pull policy updates, or do other things that require contact with the Fleet Server.
### Issue:
This problem matters because it makes it hard to detect when the Elastic Agent itself enters a state that it can no longer properly function in. While I can see the agent `Offline` in Kibana because it hasn't checked in recently. If I have a tool like Puppet which periodically checks agents that are unhealthy and attempts to fix them, I run into an issue, where Puppet wouldn't be able to detect this issue, and therefore not automatically fix the issue, requiring manual intervention.
Contributor guide
Research direction
Start with fleet/fleet_gateway.go around line 205 and reproduce the disconnected Fleet Server state using the Elastic Agent status command. Trace how connection errors affect the top-level status, then verify that the status reports UNHEALTHY when the agent can no longer communicate with Fleet Server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100