buildkite / buildkite/agent-stack-k8s
[FEATURE] opt-in flag to preserve drain-on-SIGTERM behaviour (v3.110.0+)
- Dominant language
- Go
- Stars
- 110
- Forks
- 57
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 5
Description
## Is your feature request related to a problem? Please describe.
Prior to v3.110.0, agent-stack-k8s handled SIGTERM by draining gracefully. The agent waited for the current job to finish before disconnecting.
We've heard from our Buildkite contact that starting in v3.110.0, this behaviour changed. The agent now immediately cancels the running job on SIGTERM instead of draining.
This would cause issues for us.
We run Buildkite agents on Kubernetes with `Karpenter` using the `WhenEmptyOrUnderutilized` consolidation policy. When Karpenter identifies an underutilized node, it evicts pods via SIGTERM.
With the new v3.110.0+ behaviour, this causes running CI jobs to be cancelled rather than completing.
## Describe the solution you'd like
Add an opt-in flag or config option to agent-stack-k8s to restore the previous drain-on-SIGTERM behaviour: when a job pod receives SIGTERM, the agent should attempt to finish the current job rather than cancelling it immediately.
## Describe alternatives you've considered
N/A
## Affiliation (optional)
Faire
## Additional context
Conversation within our Buildkite Slack channel: https://faire-wholesale.slack.com/archives/C06HR118GSW/p1782334084676339.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by locating agent-stack-k8s configuration and SIGTERM handling, then trace how a job pod responds to termination. Done means an opt-in setting restores graceful draining while the default behavior remains unchanged, with coverage for both paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100