buildkite / buildkite/agent-stack-k8s

[BUG] Controller can panic during informer replay before job watcher context is initialized

Open
#951 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
110
Forks
57
Avg merge
3d 8h
Merged PRs (30d)
5

Description

# [BUG] Controller can panic during informer replay before job watcher context is initialized

## Describe the bug

agent-stack-k8s v0.49.0 can panic during controller startup while replaying existing Kubernetes Jobs.

The job watcher registers itself with the shared informer before assigning its event-handler context:

```go
if _, err := jobInformer.AddEventHandler(w); err != nil {
return err
}
w.resourceEventHandlerCtx = ctx
```

The shared informer has already been started and synchronized by earlier controller components. Adding the job watcher can therefore replay cached Jobs immediately. `OnAdd` can call `runChecks` with a nil context, and a finished Job with a job-acquisition-token Secret reaches `secrets.Get(ctx, ...)`, which panics inside client-go.

We observed three consecutive controller exits during startup with the same panic. Kubernetes restarted the controller and a subsequent start succeeded.

## To Reproduce

1. Deploy agent-stack-k8s v0.49.0 with job acquisition tokens enabled.
2. Leave completed agent-stack Jobs with the job-acquisition-token Secret annotation in the namespace.
3. Restart the controller after the shared Job informer has cached those Jobs.
4. During cached-object replay, `jobWatcher.OnAdd` may run before `resourceEventHandlerCtx` is assigned.
5. Token cleanup calls the Kubernetes client with a nil context and the controller panics.

## Expected behavior

Informer callbacks always receive a valid context. Startup against existing completed Jobs does not panic, and their job-acquisition-token Secrets are cleaned up normally.

## Environment

- agent-stack-k8s version: v0.49.0
- controller image: `ghcr.io/buildkite/agent-stack-k8s/controller:0.49.0`
- Kubernetes version: v1.35.7-gke.1027000
- Kubernetes provider: GKE
- Deployment method: official Helm chart managed through GitOps/Flux
- Job acquisition tokens: enabled

## Logs

```text
panic: runtime error: invalid memory address or nil pointer dereference

The relevant call chain was:

k8s.io/client-go/rest.(*Request).Do
scheduler.(*jobWatcher).cleanupJobAcquisitionTokenSecret
scheduler.(*jobWatcher).runChecks
scheduler.(*jobWatcher).OnAdd
```

The process exited with status 2. The same startup panic occurred three times.

## Affiliation (optional)

NVIDIA

## Additional context

The suspected race is in `RegisterInformer`:

https://github.com/buildkite/agent-stack-k8s/blob/v0.49.0/internal/controller/scheduler/job_watcher.go#L89-L112

The shared informer is started and synchronized before the job watcher is registered:

https://github.com/buildkite/agent-stack-k8s/blob/v0.49.0/internal/controller/controller.go#L211-L285

Assigning or capturing the context before calling `AddEventHandler` should close the race. A regression test should register the watcher against an already-running informer containing a completed annotated Job.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/controller/scheduler/job_watcher.go around RegisterInformer and compare its registration order with internal/controller/controller.go around informer startup and synchronization. Add a regression test using an already-running informer containing a completed annotated Job. Done means replay callbacks receive a valid context, token cleanup completes, and controller startup no longer panics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.