Distribute jobs more evenly across hosts
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
**Is your feature request related to a problem? Please describe.**
We have hosts that can have different numbers of spawned agents. The priority for these is set by the spawn ID with the [spawn-with-priority](https://buildkite.com/docs/agent/v3/cli-start#spawn-with-priority) option.
The way that priorities work now are that higher numbered priorities are used first.
If `hostA` has 1 spawned agent running and `hostB` has 3 spawned agents running, `hostB` is going to be running at least 2 or maybe 3 tests while `hostA` is sitting idle waiting for jobs to be assigned to it.
Assuming all agents are idle, the order that jobs are assigned is:
1. `hostB agent3`
2. `hostB agent2`
3. `hostA agent1` or `hostB agent1`
4. `hostA agent1` or `hostB agent1` (whichever was not given a job before)
**Describe the solution you'd like**
The next agent would be chosen based on the spawned agent utilisation of each host.
`hostA` has 1 spawned agent with 1 job running (100% utilisation)
`hostB` has 3 spawned agents with 1 job running (33% utilisation)
`hostC` has 5 spawned agents with 1 job running (20% utilisation)
The next host to be assigned work would be `hostC` because the current utilisation is the lowest. The agent on `hostC` that is given the work is determined based on the priority.
(Ideally that spawned agent prioritisation could also be flipped so `hostC agent1` would be the first to be used instead of `hostC agent5`. Having that as a configuration option would be ace! I can split that out into a separate feature request if needed.)
`hostA` has 1 spawned agent with 1 job running (100% utilisation)
`hostB` has 3 spawned agents with 1 job running (33% utilisation)
`hostC` has 5 spawned agents with 2 jobs running (40% utilisation)
Now, with `hostC` utilisation at 40%, the next host to be assigned a job would be `hostB`.
**Describe alternatives you've considered**
I've spoken with Jarryd from Buildkite about this issue, but there doesn't appear to be any existing solutions for this use case.
Setting host priority doesn't work for situations where there are, say, two agents on a host. If that host is meant to be used first due to host priority, then the same situation would occur as the original problem, where one host is doing all the work while the other is sitting idle.
**Additional context**
We set the number of spawn agents in each host's config.
There are a variety of hardware profiles for our hosts, so some can only run one agent at a time, some run 3, and we're about to start trialling hosts that should be able to run 6 or more agents 🤞
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the agent scheduling logic and its spawned-agent priority handling, then compare the behavior with the host-utilisation examples in the issue; done means jobs select the least-utilised host while preserving the configured agent priority.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ci-cd, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100