buildkite / buildkite/agent-stack-k8s

[BUG] Controllers sharing a queue repeatedly contend for the same job reservations

Open
#953 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
110
Forks
57
Avg merge
3d 8h
Merged PRs (30d)
5

Description

## Describe the bug

Two independent controllers with distinct controller IDs targeting the same Buildkite queue repeatedly received overlapping job sets and attempted to reserve the same jobs.

During a 14-minute capture, 294 reservation results were `not_reserved`. Of those 294 results, 290 corresponded to job IDs that were accepted and passed downstream by the peer controller. Successful reservations were split 443 versus 375.

The logs directly demonstrate cross-controller reservation contention. They do not establish a fairness defect, phase-lock, restart dependence, or a connection to reported queue-wait latency.

## To Reproduce

The behavior is timing-sensitive:

1. Deploy two independent controllers with distinct IDs targeting the same Buildkite queue.
2. Give them identical or similar polling configuration.
3. Feed a steady stream of jobs into the queue.
4. Compare:

* `buildkite_monitor_jobs_returned_total`
* Reservation results in both controller logs
* `buildkite_deduper_jobs_marked_running_total`
5. Correlate job IDs and look for both controllers receiving the same jobs, followed by one reserving them successfully while the peer receives `not_reserved`.

## Expected behavior

Occasional reservation races may be unavoidable, and an exact 50/50 distribution is not expected.

Occasional reservation races may be expected when multiple controllers share a queue, and an exact 50/50 distribution is not expected. It is unclear whether the repeated overlap observed here is expected behavior at this frequency.

## Environment

* agent-stack-k8s version: v0.47.0
* Kubernetes version: GKE; exact version not captured
* Deployment method: Two independent Helm installations in separate clusters
* `max-in-flight`: 150 per controller
* `job-creation-concurrency`: 50 per controller

## Logs

```
# Controller B
09:31:54.264 job processing completed agent-api-jobs-processed=2
09:31:54.264 reserving jobs count=2
09:31:54.610 job reservation completed reserved-job-count=2 not-reserved-job-count=0

# Controller A
09:31:54.482 job processing completed agent-api-jobs-processed=2
09:31:54.482 reserving jobs count=2
09:31:54.650 job reservation completed reserved-job-count=0 not-reserved-job-count=2
```

Aggregate results from the 14-minute capture:

```
Controller A: 443 reserved, 170 not_reserved
Controller B: 375 reserved, 124 not_reserved

Total not_reserved: 294
Job IDs matched to downstream handling by peer: 290
```

## Affiliation (optional)

Buildkite

## Additional context

Inspection of v0.47.0 shows:

* `internal/controller/monitor/monitor.go` drives polling with a fixed `time.NewTicker`. There is no per-poll jitter that continually varies relative timing between controllers.
* `internal/controller/controller.go` places reservation before the downstream `max-in-flight` limiter.
* `internal/controller/reserver/reserver.go` attempts to reserve the fetched jobs, splitting the received set into chunks of up to 1,000.

Reservation of the fetched set is therefore not bounded by the limiter’s current downstream capacity. These implementation details allow repeated cross-controller contention, but the available evidence does not establish why it occurred at the observed frequency or whether that frequency is expected.

One hypothesis worth testing is that the fixed polling cadence allows the controllers’ relative polling phase to remain stable, producing recurring overlap and a timing advantage during reservation. Per-poll jitter could be tested as an experiment or mitigation, but phase-lock has not been established as the root cause.

The logs were captured in a different window from the original queue-wait reports, so this issue makes no claim that reservation contention caused those waits.

If repeated reservation contention is intentional as part of the multi-controller coordination model, clarification of the expected behavior and operating characteristics would be helpful.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading internal/controller/monitor/monitor.go, internal/controller/controller.go, and internal/controller/reserver/reserver.go, then reproduce with two controllers sharing a queue. Compare buildkite_monitor_jobs_returned_total, reservation results, and buildkite_deduper_jobs_marked_running_total while correlating job IDs. Done means establishing whether the observed overlap is expected and identifying a supported mitigation or a clearly documented operating expectation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.