bazelbuild / bazelbuild/bazel

Symlinked sandbox is slow

Open
#16,711 16 comments 4 reactions 1 assignee Claimed by @metti View on GitHub
P3 team-Local-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

The symlinked sandbox is slow when there is a large number of input files (I have seen reports of actions with up to 300K)

There are a number of ways one could improve this:
1. Creating the input directories and symlinks on multiple threads (`SandboxHelpers` currently does this on one thread)
2. Traversing the Java -> C++ boundary less frequently
3. Using one symlink per large tree artifact instead of symlinking each file in it separately
4. Using io_uring on Linux for more efficient data transfer to the kernel
5. Keeping the file system created for an action around and re-using it if the same action (or a similar one) is executed again

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

_No response_

### What is the output of `bazel info release`?

_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.