bazelbuild / bazelbuild/bazel

Add features to workers to allow asset management (licenses, hardware, etc)

Open
#12,467 13 comments 0 reactions 0 assignees View on GitHub
not stale P3 team-Local-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description

Extend multiplex workers to handle the asset management use cases:

- provide ways to control time accounting so action timeouts don't need to include asset acquisition time
- provide more control over available execution slots so that job slots aren't wasted waiting on assets

I am using "assets" to mean a limited shared resource like software licenses or specialized hardware.

### Feature requests: what underlying problem are you trying to solve with this feature?

We have some actions that need access to limited shared resources such as license files or special hardware. In our case, acquiring the asset comes with considerable set-up cost which we wanted to amortize over multiple actions.

It seems like multiplex workers were the perfect tool for this. The idea was to write a worker that:

- queued up the actions
- acquired the resources and set them up
- executed the actions as the resources allowed
- released the resources when the queue was empty

In our case we need to see the action before we can acquire the resource and set it up (we are actually managing multiple kinds of resources and separate queues for each), so we can't use a simpler strategy like acquiring the resource on worker start-up and not read stdin until we have it.

However we hit a couple of snags. The first is that we wanted to do this for test actions, so we are blocked by #7595. However there are also a couple of limitations on the workers that make it not a great fit:

- When contention is high, there may be a long wait acquiring an asset. There is also a long set-up time. Unfortunately while the worker is blocked the timer is still running on the action so it risks failing due to a timeout. We want to make the timeouts useful based on the test sizes but if we have to include the worst-case asset acquisition and set-up time, they all just end up needing the maximum timeout, which is useless. We need some way control the clock in the worker so that we only start it when the action is ready to run.
- The supply of assets varies widely over time. Sometimes we can get only one, sometimes we can have dozens. We want to be able to throttle actions sent to the worker based on available assets. We can statically configure the number of concurrent actions to the worker, but if we set it high hoping there are a lot of available assets, then we can starve the rest of the build of job slots with all the jobs just waiting their turn for the scarce resources, but if we set it really low, we can unnecessarily serialize the work which can make it take much longer. We really need some way in the worker protocol to tell Bazel that we are ready to accept more actions, or that we are full and it should hold-off. Basically some kind of flow control.

### What operating system are you running Bazel on?

macOS and Linux

### What's the output of `bazel info release`?

3.5.0

### Have you found anything relevant by searching the web?

no

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are identified in the issue. Start by locating the multiplex worker protocol and related Bazel design discussions, then determine whether timeout control and worker-side flow control can be specified consistently; done requires an agreed design and implementation plan for both capabilities.

Written by the indexing model from the issue text.

Assessment

Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.