buildkite / buildkite/agent-stack-k8s
[BUG] Controller can panic during informer replay before job watcher context is initialized
- Dominant language
- Go
- Stars
- 110
- Forks
- 57
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 5
Description
# [BUG] Controller can panic during informer replay before job watcher context is initialized
## Describe the bug
agent-stack-k8s v0.49.0 can panic during controller startup while replaying existing Kubernetes Jobs.
The job watcher registers itself with the shared informer before assigning its event-handler context:
```go
if _, err := jobInformer.AddEventHandler(w); err != nil {
return err
}
w.resourceEventHandlerCtx = ctx
```
The shared informer has already been started and synchronized by earlier controller components. Adding the job watcher can therefore replay cached Jobs immediately. `OnAdd` can call `runChecks` with a nil context, and a finished Job with a job-acquisition-token Secret reaches `secrets.Get(ctx, ...)`, which panics inside client-go.
We observed three consecutive controller exits during startup with the same panic. Kubernetes restarted the controller and a subsequent start succeeded.
## To Reproduce
1. Deploy agent-stack-k8s v0.49.0 with job acquisition tokens enabled.
2. Leave completed agent-stack Jobs with the job-acquisition-token Secret annotation in the namespace.
3. Restart the controller after the shared Job informer has cached those Jobs.
4. During cached-object replay, `jobWatcher.OnAdd` may run before `resourceEventHandlerCtx` is assigned.
5. Token cleanup calls the Kubernetes client with a nil context and the controller panics.
## Expected behavior
Informer callbacks always receive a valid context. Startup against existing completed Jobs does not panic, and their job-acquisition-token Secrets are cleaned up normally.
## Environment
- agent-stack-k8s version: v0.49.0
- controller image: `ghcr.io/buildkite/agent-stack-k8s/controller:0.49.0`
- Kubernetes version: v1.35.7-gke.1027000
- Kubernetes provider: GKE
- Deployment method: official Helm chart managed through GitOps/Flux
- Job acquisition tokens: enabled
## Logs
```text
panic: runtime error: invalid memory address or nil pointer dereference
The relevant call chain was:
k8s.io/client-go/rest.(*Request).Do
scheduler.(*jobWatcher).cleanupJobAcquisitionTokenSecret
scheduler.(*jobWatcher).runChecks
scheduler.(*jobWatcher).OnAdd
```
The process exited with status 2. The same startup panic occurred three times.
## Affiliation (optional)
NVIDIA
## Additional context
The suspected race is in `RegisterInformer`:
https://github.com/buildkite/agent-stack-k8s/blob/v0.49.0/internal/controller/scheduler/job_watcher.go#L89-L112
The shared informer is started and synchronized before the job watcher is registered:
https://github.com/buildkite/agent-stack-k8s/blob/v0.49.0/internal/controller/controller.go#L211-L285
Assigning or capturing the context before calling `AddEventHandler` should close the race. A regression test should register the watcher against an already-running informer containing a completed annotated Job.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in internal/controller/scheduler/job_watcher.go around RegisterInformer and compare its registration order with internal/controller/controller.go around informer startup and synchronization. Add a regression test using an already-running informer containing a completed annotated Job. Done means replay callbacks receive a valid context, token cleanup completes, and controller startup no longer panics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100