elastic / elastic/fleet-server
Changing from Elasticsearch to Logstash output and back causes agents to go offline
- Dominant language
- Go
- Stars
- 113
- Forks
- 117
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 112
Description
- Version: 8.6.1
- Operating System: Windows 10 and Windows Server 2019
- Discuss Forum URL:
## - Steps to Reproduce:
* Navigate to Fleet -> Agents Policy tab
* Select the fleet policy
* Click on the fleet policy settings
* Change Fleet Policy output for integrations to Logstash
* Change output for Agent monitoring to Logstash
* Save changes
Upon failure :
* Remove the problematic fleet server
* Update the Fleet Server policy output for integrations and monitoring under settings to point back to Elastic Search instead of logstash
* Setup and add a new fleet server
* Add the Fleet server to Kibana using the old Fleet Policy
* Check the agents status
Another symptoms that we seeing in the log agent that are offline are reporting
```
possible transient error during checking with fleet-server , retrying
```
I cant provide the logs .
I know we need to upgrade to 8.7 or even 8.8 and we are planning to do so to resolve the problems around this bug report elastic/elastic-agent#2316 . However, I suspect there is a also problem with the recovery functions. Where after we stood up a new fleet server following the changes made above bu most of the agents remain unhealthy. The majority of agents are showing as offline which is also may may be discussed on elastic/elastic-agent#2554
A reboot on some of the systems fixed the issue because the service get hung after a service restart .However Its not possible to send a mass reboot command to all systems . Multiple agents remain to be in the unhealthy status.
Its worth mentioning that I opened an enhancement request to add the functions of [2523](https://github.com/elastic/fleet-server/issues/2523)
Contributor guide
Assessment
This issue has not been assessed yet.