elastic / elastic/elastic-agent
[beats receivers] Collector sub-process shutdown timeout needs to be configurable and contextual
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
- Relates https://github.com/elastic/elastic-agent/issues/10632
- Relates https://github.com/elastic/elastic-agent/pull/10650
The coordinator waits up to 5s for the managers to exit:
https://github.com/elastic/elastic-agent/blob/1237cc38ae24e70791213bde4bd2082a88cde744/internal/pkg/agent/application/coordinator/coordinator.go#L231-L233
In https://github.com/elastic/elastic-agent/pull/10650 a new collector stop timeout was introduced at 3s long to guarantee the collector sub-process was killed without failing to wait for it, avoiding a defunct zombie process.
Similarly, the default stop timeout for other sub-processes is 30s, and it is by coincidence that existing sub-processes exit faster than this: https://github.com/elastic/elastic-agent/blob/1237cc38ae24e70791213bde4bd2082a88cde744/pkg/core/process/config.go#L22
The amount of time to wait before killing sub-processes needs to be configurable by the user, and in most cases set to a longer value like the existing 30s (like the collector timeout originally was) to allow for graceful shutdown and for final data to be shipped. The collector is currently special cased to a shorter timeout because it has to restart whenever it's configuration changes as of the time of writing.
However, in certain circumstances waiting this long is inconveniencing, for example when enrolling an Elastic Agent to Fleet it restarts itself and would incur this shutdown delay. In this case the existing configuration is being discarded and agent can restart immediately.
## Acceptance Criteria
- The shutdown timeout for all sub-processes needs to be configurable by the user in elastic-agent.yml.
- Cases where graceful shutdown is not necessary should use the lowest timeout possible by default. For example, when enrolling Elastic Agent into a new agent policy.
- The increased shutdown delay must not cause unexpected impact to other parts of agent that re-execute as part of its implementation, particularly the upgrade process.
Contributor guide
Research direction
Start with the shutdown handling in internal/pkg/agent/application/coordinator/coordinator.go and the default process settings in pkg/core/process/config.go, then review the related issue 10632 and pull request 10650. Trace how subprocesses are restarted during enrollment and upgrades; done means elastic-agent.yml can configure shutdown timeouts while graceful cases and re-execution paths behave as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100