elastic / elastic/elastic-agent

[beats receivers] Collector sub-process shutdown timeout needs to be configurable and contextual

Open
#10,786 1 comment 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 23h
Merged PRs (30d)
312

Description

- Relates https://github.com/elastic/elastic-agent/issues/10632
- Relates https://github.com/elastic/elastic-agent/pull/10650

The coordinator waits up to 5s for the managers to exit:

https://github.com/elastic/elastic-agent/blob/1237cc38ae24e70791213bde4bd2082a88cde744/internal/pkg/agent/application/coordinator/coordinator.go#L231-L233

In https://github.com/elastic/elastic-agent/pull/10650 a new collector stop timeout was introduced at 3s long to guarantee the collector sub-process was killed without failing to wait for it, avoiding a defunct zombie process.

Similarly, the default stop timeout for other sub-processes is 30s, and it is by coincidence that existing sub-processes exit faster than this: https://github.com/elastic/elastic-agent/blob/1237cc38ae24e70791213bde4bd2082a88cde744/pkg/core/process/config.go#L22

The amount of time to wait before killing sub-processes needs to be configurable by the user, and in most cases set to a longer value like the existing 30s (like the collector timeout originally was) to allow for graceful shutdown and for final data to be shipped. The collector is currently special cased to a shorter timeout because it has to restart whenever it's configuration changes as of the time of writing.

However, in certain circumstances waiting this long is inconveniencing, for example when enrolling an Elastic Agent to Fleet it restarts itself and would incur this shutdown delay. In this case the existing configuration is being discarded and agent can restart immediately.

## Acceptance Criteria

- The shutdown timeout for all sub-processes needs to be configurable by the user in elastic-agent.yml.
- Cases where graceful shutdown is not necessary should use the lowest timeout possible by default. For example, when enrolling Elastic Agent into a new agent policy.
- The increased shutdown delay must not cause unexpected impact to other parts of agent that re-execute as part of its implementation, particularly the upgrade process.

Contributor guide

Open the contributing guide

Research direction

Start with the shutdown handling in internal/pkg/agent/application/coordinator/coordinator.go and the default process settings in pkg/core/process/config.go, then review the related issue 10632 and pull request 10650. Trace how subprocesses are restarted during enrollment and upgrades; done means elastic-agent.yml can configure shutdown timeouts while graceful cases and re-execution paths behave as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.