bazelbuild / bazelbuild/bazel

Efficient way to relocate files in runfiles

Open
#23,728 4 comments 0 reactions 0 assignees View on GitHub
team-Core type: feature request untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

There are situations where having more granular control over the layout of files and directories in runfiles is beneficial.

## Python context

Specifically, for rules_python, when we import third_party wheels (archives) from indexes like PyPI (Python Package Index), we would ideally like to expand the archives into a directory called `site-packages`. This is because outside the bazel context, packages are expanded as siblings inside `site-packages`.

This creates a major impedance mismatch between bazel and archives or foreign packages brought in from other ecosystems. It truly breaks assumptions in breaking ways in Python. One example (there are many others): NVIDIA wheels are expected to be installed as siblings of site-packages and they even have relative `RPATH` inside their `.so`. There are no built-in mechanisms to solve this cleanly in Python or bazel at the moment. See https://github.com/bazelbuild/rules_python/issues/2156

Here is a typical example of the contents of `site-packages` for `pip install requests` where you can see the standard layout when not using bazel.

```
tree -L 1 .venv/lib/python3.9/site-packages/ 20:49:03
.venv/lib/python3.9/site-packages/
├── _distutils_hack
├── certifi
├── certifi-2024.8.30.dist-info
├── charset_normalizer
├── charset_normalizer-3.3.2.dist-info
├── distutils-precedence.pth
├── idna
├── idna-3.10.dist-info
├── pip
├── pip-22.0.4.dist-info
├── pkg_resources
├── requests
├── requests-2.32.3.dist-info
├── setuptools
├── setuptools-58.1.0.dist-info
├── urllib3
└── urllib3-2.2.3.dist-info
```

## Other contexts

This is more than just a Python problem. Node has similar challenges with `node_modules`.

cc @rickeylev @fmeum

### Which category does this issue belong to?

_No response_

### What underlying problem are you trying to solve with this feature?

_No response_

### Which operating system are you running Bazel on?

All

### What is the output of `bazel info release`?

N/A

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

N/A

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
N/A
```

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

The request concerns Bazel runfiles layout, especially expanding Python wheels under site-packages, and references rules_python issue #2156. Start by reviewing runfiles handling and that linked issue; done means a supported way to relocate files or directories while preserving the required package layout, with coverage for the stated Python use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.