buildkite / buildkite/agent

Add `healthcheck` Agent Lifecycle Hook that runs between jobs

Open
#1,111 6 comments 0 reactions 0 assignees View on GitHub
agent health hook
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

`buildkite-agent` has a `disconnect-after-idle-timeout` setting which causes it to shut down after an idle period.

It would be nice if there was an option to disconnect / shut down the agent if a healthcheck fails. We run an elastic autoscaled pool of agents, and shutting down in the case of a problem would allow the autoscaling group to replace unhealthy agents.

Currently when an agent has its disk fill up for example then all jobs allocated to it will fail, but it will still keep accepting jobs regardless. These often require manual intervention to kill the instance and let the pool recover. Having the agent stop accepting jobs and shut down when available disk space is low would allow them to be replaced with clean agents without manual intervention.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.