agent-shutdown hook should trigger on fatal errors
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
**Is your feature request related to a problem? Please describe.**
buildkite workers that encounter network-related errors do not trigger the `agent-shutdown` hook properly.
For example, when the network dies on my agent, and the agent is unable to POST to `https://agent.buildkite.com/v3/register` 30x in a row, the agent dies but does not trigger the `agent-shutdown` hook. I have important cleanup code that must be run after every `buildkite-agent start` command, and so failing to run the `agent-shutdown` hook is a big issue.
**Describe the solution you'd like**
The `agent-shutdown` hook should always be run, as long as the arguments provided to `buildkite-agent` are valid.
**Describe alternatives you've considered**
I can wrap my `buildkite-agent start` invocation in a script that runs the cleanup code there instead, but I feel this rather defeats the purpose of having an `agent-shutdown` hook.
**Additional context**
In our case, we use the `agent-shutdown` hook, combined with `--disconnect-after-job` and to reboot VMs, which are configured to reset to a pristine state on reboot. When the agent fails to connect due internet issues and fails to run the `agent-shutdown` hook, the VMs get stuck. While I will likely implement a fallback rebooting option, it would be nice if `agent-shutdown` was more consistently triggered.
Contributor guide
Assessment
This issue has not been assessed yet.