elastic / elastic/elastic-agent
Custom Windows Event Logs integration configuration can cause Elastic Agent crashing
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 303
Description
In Custom Windows Event Logs integration configuration there is a field to specify "Dataset name". Field is initialized with "winlog.winlog" value but it can be left empty. When you leave "Dataset name" empty and assign policy with this integration Elastic Agent starts crashing.
Windows system log repeating during faulty policy assigned to ELastic Agent:
```
The Elastic Agent service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 1000 milliseconds:
```
Elastic Agent logs
```
Jun 27, 2022 @ 15:51:29.191 Fleet gateway is stopped
Jun 27, 2022 @ 15:51:27.823 2022-06-27T15:51:27+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:27.823 2022-06-27T15:51:27+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:27.823 2022-06-27T15:51:27+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:27.283 Starting stats endpoint
Jun 27, 2022 @ 15:51:27.283 Agent is starting
Jun 27, 2022 @ 15:51:27.283 Metrics endpoint listening on: \\.\pipe\elastic-agent (configured: npipe:///elastic-agent)
Jun 27, 2022 @ 15:51:26.672 Updating internal state
Jun 27, 2022 @ 15:51:26.656 2022-06-27T15:51:26+02:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:26.124 operation 'operation-install' skipped for metricbeat.8.2.0
Jun 27, 2022 @ 15:51:25.230 2022-06-27T15:51:25+02:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:24.416 operation 'operation-install' skipped for filebeat.8.2.0
Jun 27, 2022 @ 15:51:23.380 2022-06-27T15:51:23+02:00 - message: Application: filebeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:22.983 operation 'operation-install' skipped for filebeat.8.2.0
Jun 27, 2022 @ 15:51:21.881 2022-06-27T15:51:21+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:21.459 operation 'operation-install' skipped for osquerybeat.8.2.0
Jun 27, 2022 @ 15:51:18.559 New State ID is QrXHC_do
Jun 27, 2022 @ 15:51:18.559 Converging state requires execution of 3 step(s)
Jun 27, 2022 @ 15:51:18.302 restoring current policy from disk
Jun 27, 2022 @ 15:51:18.298 Docker provider skipped, unable to connect: protocol not available
Jun 27, 2022 @ 15:51:17.343 capabilities file not found in C:\Program Files\Elastic\Agent\capabilities.yml
Jun 27, 2022 @ 15:51:17.340 Agent is managed by Fleet
Jun 27, 2022 @ 15:51:17.338 Detecting execution mode
Jun 27, 2022 @ 15:51:17.096 APM instrumentation disabled
Jun 27, 2022 @ 15:51:15.023 Fleet gateway is stopped
Jun 27, 2022 @ 15:51:13.085 Starting stats endpoint
Jun 27, 2022 @ 15:51:13.085 Agent is starting
Jun 27, 2022 @ 15:51:13.085 Metrics endpoint listening on: \\.\pipe\elastic-agent (configured: npipe:///elastic-agent)
Jun 27, 2022 @ 15:51:12.430 Updating internal state
Jun 27, 2022 @ 15:51:09.525 2022-06-27T15:51:09+02:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:09.525 2022-06-27T15:51:09+02:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:09.523 2022-06-27T15:51:09+02:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:07.342 2022-06-27T15:51:07+02:00 - message: Application: filebeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:07.342 2022-06-27T15:51:07+02:00 - message: Application: filebeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:07.342 2022-06-27T15:51:07+02:00 - message: Application: filebeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:02.744 2022-06-27T15:51:02+02:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:02.139 operation 'operation-install' skipped for metricbeat.8.2.0
Jun 27, 2022 @ 15:51:01.357 2022-06-27T15:51:01+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:01.355 2022-06-27T15:51:01+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:01.355 2022-06-27T15:51:01+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:00.767 2022-06-27T15:51:00+02:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:51:00.358 operation 'operation-install' skipped for filebeat.8.2.0
Jun 27, 2022 @ 15:50:56.975 2022-06-27T15:50:56+02:00 - message: Application: filebeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:50:56.699 operation 'operation-install' skipped for filebeat.8.2.0
Jun 27, 2022 @ 15:50:55.982 2022-06-27T15:50:55+02:00 - message: Application: osquerybeat--8.2.0[e8c6e830-0751-4b61-89e2-5e9ff16b937e]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
Jun 27, 2022 @ 15:50:55.736 operation 'operation-install' skipped for osquerybeat.8.2.0
Jun 27, 2022 @ 15:50:51.795 New State ID is QrXHC_do
Jun 27, 2022 @ 15:50:51.795 Converging state requires execution of 3 step(s)
Jun 27, 2022 @ 15:50:51.672 restoring current policy from disk
Jun 27, 2022 @ 15:50:51.669 Docker provider skipped, unable to connect: protocol not available
Jun 27, 2022 @ 15:50:51.289 Agent is managed by Fleet
Jun 27, 2022 @ 15:50:51.289 capabilities file not found in C:\Program Files\Elastic\Agent\capabilities.yml
Jun 27, 2022 @ 15:50:51.288 Detecting execution mode
Jun 27, 2022 @ 15:50:51.223 APM instrumentation disabled
Jun 27, 2022 @ 15:50:49.482 Fleet gateway is stopped
```
In filebeat and osquery logs these lines are repeating (about 1000 times per minute):
```
elastic-agent-client got error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: internal error"
elastic-agent-client got error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:6789: connectex: No connection could be made because the target machine actively refused it.
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
```
After fixing integration policy on Windows system filebeat, osquery and meatricbeat processes are consuming 100% od CPU and killing those processes is nessecery.
Whole stack (Elasticsearch, Kibana, Fleet, Elastic Agent) at version 8.2.0. Except Elastic Agent running on Windows OS all other components are deployed as Docker containers.
Contributor guide
Assessment
This issue has not been assessed yet.