elastic / elastic/fleet-server

In self-managed elastic, the agent's logs are not visible in kibana

Open
#3,415 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
113
Forks
117
Avg merge
1d 16h
Merged PRs (30d)
112

Description

Hello, everyone.
I set up a cluster with three Linux servers using elastic version 8.11.1. The information for each server is as follows: Server 1 ip-10.150.3.12 elasticsearch+kibana, server 2 ip-10.150.3.15 elasticsearch+elastic agent, and server 3 ip 10.150.3.17 elasticsearch+fleetserver. They are self-managed clusters and are configured with TSL, using self-generated certificates. When installing fleetserver and agent, I didn't do any configuration for Elasti-agent.yml, just tar and install. fleetserver and agent are up and running, but in kibana, don't see logs. May I ask where I need to troubleshoot the problem next?
I uploaded my diagnostics log at the end, hoping to help with this problem. Thank you so much!

![1](https://github.com/elastic/fleet-server/assets/75662809/a5afec87-ae96-457b-a466-b878433807c8)
![2](https://github.com/elastic/fleet-server/assets/75662809/74e67662-d36c-44fe-be35-acf65ff90605)
![3](https://github.com/elastic/fleet-server/assets/75662809/eeb24f37-93b4-4718-9cd8-39a988a50339)
![4](https://github.com/elastic/fleet-server/assets/75662809/ecc74250-462c-42a9-af5a-d29e917e0289)
![5](https://github.com/elastic/fleet-server/assets/75662809/3667381a-f899-452f-b77b-1791eda39ebc)
![6](https://github.com/elastic/fleet-server/assets/75662809/6689d4de-ab44-4b77-b76a-f115919a4c6a)

```
sudo ./elastic-agent install --url=https://10.150.3.17:8220 \
> --fleet-server-es=https://10.150.3.17:9200 \
> --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTE5NTgxNDE1NjE6Zi1LN0hJQVZUSUdlMjJZejJ5QU9YQQ \
> --fleet-server-policy=5906c780-ec1c-11ee-9cd9-39245a862599 \
> --fleet-server-es-ca-trusted-fingerprint=BC8F64D445A43597BE161251826AAA1B302C5C76BDE744C479E36C8481E10704 \
> --certificate-authorities=/home/elastic/downloads/beats/elastic-agent/fleet-server/ca/ca.crt \
> --fleet-server-cert=/home/elastic/downloads/beats/elastic-agent/fleet-server/ca/fleet-server.crt \
> --fleet-server-cert-key=/home/elastic/downloads/beats/elastic-agent/fleet-server/ca/fleet-server.key \
> --fleet-server-port=8220 \
> --fleet-server-es-insecure
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
Copying files......................... DONE
Installing service... DONE
Starting service... DONE
Enrolling Elastic Agent with Fleet...................{"log.level":"info","@timestamp":"2024-04-01T15:56:31.046+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":774},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
...............{"log.level":"info","@timestamp":"2024-04-01T15:56:35.050+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":788},"message":"Fleet Server - Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T15:56:35.276+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Starting enrollment to URL: https://10.150.3.17:8220/","ecs.version":"1.6.0"}
....{"log.level":"info","@timestamp":"2024-04-01T15:56:36.315+0800","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":277},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
DONE
Elastic Agent has been successfully installed.
```

```

[root@10.150.3.17 elastic-agent-03ef9d]# ./elastic-agent status
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
└─ status: (HEALTHY) Running
[root@10.150.3.17 elastic-agent-03ef9d]# ./elastic-agent status --output=json
{
"info": {
"id": "2ca3cb94-9bb4-43a8-b4b5-44a312a45de6",
"version": "8.11.1",
"commit": "03ef9d8d6a894af61a77d78983a735d263b414ac",
"build_time": "2023-11-11 13:29:06 +0000 UTC",
"snapshot": false
},
"state": 2,
"message": "Running",
"components": [
{
"id": "beat/metrics-monitoring",
"name": "beat/metrics",
"state": 2,
"message": "Healthy: communicating with pid '47052'",
"units": [
{
"unit_id": "beat/metrics-monitoring",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "beat/metrics-monitoring-metrics-monitoring-beats",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"version": "8.11.1",
"meta": {
"build_time": "2023-11-10 19:26:08 +0000 UTC",
"commit": "19c8672c0a5bf2fb15648c0caf62d985af5a987f"
}
}
},
{
"id": "filestream-monitoring",
"name": "filestream",
"state": 2,
"message": "Healthy: communicating with pid '47041'",
"units": [
{
"unit_id": "filestream-monitoring",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "filestream-monitoring-filestream-monitoring-agent",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"version": "8.11.1",
"meta": {
"build_time": "2023-11-10 19:23:39 +0000 UTC",
"commit": "19c8672c0a5bf2fb15648c0caf62d985af5a987f"
}
}
},
{
"id": "fleet-server-default",
"name": "fleet-server",
"state": 2,
"message": "Healthy: communicating with pid '46959'",
"units": [
{
"unit_id": "fleet-server-default",
"unit_type": 1,
"state": 2,
"message": "Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599"
},
{
"unit_id": "fleet-server-default-fleet-server-fleet_server-0c1e5097-b750-400f-8983-0c1fec8b58c8",
"unit_type": 0,
"state": 2,
"message": "Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599"
}
],
"version_info": {
"name": "fleet-server",
"version": "8.11.1",
"meta": {
"build_time": "2023-11-09 06:45:02 +0000 UTC",
"commit": "8a5688dc"
}
}
},
{
"id": "http/metrics-monitoring",
"name": "http/metrics",
"state": 2,
"message": "Healthy: communicating with pid '47063'",
"units": [
{
"unit_id": "http/metrics-monitoring",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "http/metrics-monitoring-metrics-monitoring-agent",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"version": "8.11.1",
"meta": {
"build_time": "2023-11-10 19:26:08 +0000 UTC",
"commit": "19c8672c0a5bf2fb15648c0caf62d985af5a987f"
}
}
},
{
"id": "log-default",
"name": "log",
"state": 2,
"message": "Healthy: communicating with pid '47013'",
"units": [
{
"unit_id": "log-default",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "log-default-logfile-system-e6016c3e-b207-40b3-a6dc-1f9a3cd1bec1",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"version": "8.11.1",
"meta": {
"build_time": "2023-11-10 19:23:39 +0000 UTC",
"commit": "19c8672c0a5bf2fb15648c0caf62d985af5a987f"
}
}
},
{
"id": "system/metrics-default",
"name": "system/metrics",
"state": 2,
"message": "Healthy: communicating with pid '47025'",
"units": [
{
"unit_id": "system/metrics-default",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "system/metrics-default-system/metrics-system-e6016c3e-b207-40b3-a6dc-1f9a3cd1bec1",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"version": "8.11.1",
"meta": {
"build_time": "2023-11-10 19:26:08 +0000 UTC",
"commit": "19c8672c0a5bf2fb15648c0caf62d985af5a987f"
}
}
}
],
"FleetState": 2,
"FleetMessage": "Connected"
}

```
```

{"log.level":"info","@timestamp":"2024-04-01T07:56:32.352Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:32.352Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:32.878Z","message":"Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-04-01T07:56:32.878Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":558},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-04-01T07:56:32.878Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":558},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: 5906c780-ec1c-11ee-9cd9-39245a862599; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:35.505Z","message":"Checking enrollment key from database j85c2GqpRw-x2JEQAmEHjQ","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"mod":"enroll","fleet.enroll.apikey.id":"oQQ9f44B6eQfthdqM8OM","ecs.version":"1.6.0","service.name":"fleet-server","http.request.id":"01HTCAHZ9966N7HWFVQZV7ZBD3","server.address":"","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:35.758Z","message":"Found enrollment key b1FROWY0NEI2ZVFmdGhkcU04T006ajg1YzJHcXBSdy14MkpFUUFtRUhqUQ==","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.type":"fleet-server","server.address":"","mod":"enroll","fleet.enroll.apikey.id":"oQQ9f44B6eQfthdqM8OM","service.name":"fleet-server","http.request.id":"01HTCAHZ9966N7HWFVQZV7ZBD3","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:36.051Z","message":"Elastic Agent successfully enrolled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"mod":"enroll","fleet.access.apikey.id":"7aGomI4B2evC5-sW_pEv","event.duration":549099726,"service.name":"fleet-server","http.request.id":"01HTCAHZ9966N7HWFVQZV7ZBD3","fleet.enroll.apikey.id":"oQQ9f44B6eQfthdqM8OM","fleet.agent.id":"2ca3cb94-9bb4-43a8-b4b5-44a312a45de6","service.type":"fleet-server","server.address":"","http.response.body.bytes":1134,"ecs.version":"1.6.0","fleet.policy.id":"5906c780-ec1c-11ee-9cd9-39245a862599","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:36.314Z","log.origin":{"file.name":"cmd/run.go","file.line":302},"message":"reexec shutdown channel triggered","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:36.315Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0","ecs.version":"1.6.0"}
```
```

{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"Uptime: 3.097177795s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":73,"file.name":"api/server.go"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"api","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.817Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":531,"file.name":"instance/beat.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.983Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":80}},"total":{"ticks":210,"time":{"ms":210},"value":210},"user":{"ticks":130,"time":{"ms":130}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":10},"info":{"ephemeral_id":"a7add003-1f25-4a1e-97bd-84a95e33eb20","name":"metricbeat","uptime":{"ms":3090},"version":"8.11.1"},"memstats":{"gc_next":47007672,"memory_alloc":37410120,"memory_sys":58586376,"memory_total":52203896,"rss":90165248},"runtime":{"goroutines":29}},"libbeat":{"config":{"module":{"running":2,"starts":2,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":1,"dropped":0,"failed":1,"filtered":0,"published":1,"retry":0,"total":2},"queue":{"acked":0,"max_events":4096}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":8},"load":{"1":0.67,"15":0.63,"5":0.62,"norm":{"1":0.0838,"15":0.0788,"5":0.0775}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Uptime: 3.092760384s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-01T07:56:30.985Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":531,"file.name":"instance/beat.go"},"ecs.version":"1.6.0"}
```

[elastic-agent-diagnostics-2024-04-01T08-21-38Z-00.zip](https://github.com/elastic/fleet-server/files/14821484/elastic-agent-diagnostics-2024-04-01T08-21-38Z-00.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.