elastic / elastic/elastic-agent

[Elastic Agent] Fleet managed Elastic Agent stays healthy when it can't connect to Fleet Server

Open
#87 4 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 20h
Merged PRs (30d)
303

Description

- Version: 7.16.2
- Operating System: CentOS 7 (but issue can probably happen with any Fleet managed Agent)
- Steps to Reproduce:
1. Install Fleet Managed Elastic Agent on a system. (.tar.gz install)
2. Wait for the Agent to register and become healthy
3. Interrupt connection between Elastic Agent and Fleet Server
4. Wait for Elastic Agent to start reporting connection error logs:
```json
{"log.level":"error","@timestamp":"2022-02-01T15:23:52.311Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://fleet-server.example.com:8220/api/fleet/agents//checkin?\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","ecs.version":"1.6.0"}
```
5. Check Elastic Agent Status, and see that it is healthy:
```bash
sudo /opt/Elastic/Agent/elastic-agent status
Status: HEALTHY
Message: (no message)
Applications:
* endpoint-security (HEALTHY)
Protecting with policy {2dc16c0f-ebc4-400a-bd5a-7afacc5e4370}
* filebeat (HEALTHY)
Running
* metricbeat (HEALTHY)
Running
* filebeat_monitoring (HEALTHY)
Running
* metricbeat_monitoring (HEALTHY)
Running
* osquerybeat (HEALTHY)
Running
```

### Expected:

I'd expect the top level status to be UNHEALTHY as the Agent can no longer talk to Fleet server and therefore no longer pull policy updates, or do other things that require contact with the Fleet Server.

### Issue:

This problem matters because it makes it hard to detect when the Elastic Agent itself enters a state that it can no longer properly function in. While I can see the agent `Offline` in Kibana because it hasn't checked in recently. If I have a tool like Puppet which periodically checks agents that are unhealthy and attempts to fix them, I run into an issue, where Puppet wouldn't be able to detect this issue, and therefore not automatically fix the issue, requiring manual intervention.

Contributor guide

Open the contributing guide

Research direction

Start with fleet/fleet_gateway.go around line 205 and reproduce the disconnected Fleet Server state using the Elastic Agent status command. Trace how connection errors affect the top-level status, then verify that the status reports UNHEALTHY when the agent can no longer communicate with Fleet Server.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.