Efficient way to relocate files in runfiles
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
There are situations where having more granular control over the layout of files and directories in runfiles is beneficial.
## Python context
Specifically, for rules_python, when we import third_party wheels (archives) from indexes like PyPI (Python Package Index), we would ideally like to expand the archives into a directory called `site-packages`. This is because outside the bazel context, packages are expanded as siblings inside `site-packages`.
This creates a major impedance mismatch between bazel and archives or foreign packages brought in from other ecosystems. It truly breaks assumptions in breaking ways in Python. One example (there are many others): NVIDIA wheels are expected to be installed as siblings of site-packages and they even have relative `RPATH` inside their `.so`. There are no built-in mechanisms to solve this cleanly in Python or bazel at the moment. See https://github.com/bazelbuild/rules_python/issues/2156
Here is a typical example of the contents of `site-packages` for `pip install requests` where you can see the standard layout when not using bazel.
```
tree -L 1 .venv/lib/python3.9/site-packages/ 20:49:03
.venv/lib/python3.9/site-packages/
├── _distutils_hack
├── certifi
├── certifi-2024.8.30.dist-info
├── charset_normalizer
├── charset_normalizer-3.3.2.dist-info
├── distutils-precedence.pth
├── idna
├── idna-3.10.dist-info
├── pip
├── pip-22.0.4.dist-info
├── pkg_resources
├── requests
├── requests-2.32.3.dist-info
├── setuptools
├── setuptools-58.1.0.dist-info
├── urllib3
└── urllib3-2.2.3.dist-info
```
## Other contexts
This is more than just a Python problem. Node has similar challenges with `node_modules`.
cc @rickeylev @fmeum
### Which category does this issue belong to?
_No response_
### What underlying problem are you trying to solve with this feature?
_No response_
### Which operating system are you running Bazel on?
All
### What is the output of `bazel info release`?
N/A
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
N/A
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
N/A
```
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
The request concerns Bazel runfiles layout, especially expanding Python wheels under site-packages, and references rules_python issue #2156. Start by reviewing runfiles handling and that linked issue; done means a supported way to relocate files or directories while preserving the required package layout, with coverage for the stated Python use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, python
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100