buildkite / buildkite/agent-stack-k8s

[FEATURE] opt-in flag to preserve drain-on-SIGTERM behaviour (v3.110.0+)

Open
#903 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
110
Forks
57
Avg merge
3d 8h
Merged PRs (30d)
5

Description

## Is your feature request related to a problem? Please describe.
Prior to v3.110.0, agent-stack-k8s handled SIGTERM by draining gracefully. The agent waited for the current job to finish before disconnecting.

We've heard from our Buildkite contact that starting in v3.110.0, this behaviour changed. The agent now immediately cancels the running job on SIGTERM instead of draining.

This would cause issues for us.
We run Buildkite agents on Kubernetes with `Karpenter` using the `WhenEmptyOrUnderutilized` consolidation policy. When Karpenter identifies an underutilized node, it evicts pods via SIGTERM.
With the new v3.110.0+ behaviour, this causes running CI jobs to be cancelled rather than completing.

## Describe the solution you'd like
Add an opt-in flag or config option to agent-stack-k8s to restore the previous drain-on-SIGTERM behaviour: when a job pod receives SIGTERM, the agent should attempt to finish the current job rather than cancelling it immediately.

## Describe alternatives you've considered
N/A

## Affiliation (optional)
Faire

## Additional context
Conversation within our Buildkite Slack channel: https://faire-wholesale.slack.com/archives/C06HR118GSW/p1782334084676339.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by locating agent-stack-k8s configuration and SIGTERM handling, then trace how a job pod responds to termination. Done means an opt-in setting restores graceful draining while the default behavior remains unchanged, with coverage for both paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.