aspect-build / aspect-build/rules_py
[FR]: More granular `py_image_layer` behavior
- Dominant language
- Starlark
- Stars
- 145
- Forks
- 97
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 71
Description
Today `py_image_layer` takes the `.runfiles` tree from a `py_binary` including the new `py_venv_*` rules and by default attempts to break that one input filegroup into three layers.
1. The "interpreter", assuming that the user is leaning on hermetic interpreters from Astral.
2. The `site-packages` tree(s), which allows for layer reuse between images which share the same _total set_ of 3rdparty input files.
3. User files / everything else.
While this makes sense as a basic splitting strategy, it leaves much to be desired.
1. The interpreter has its own `site-packages` files which _are_ part of the interpreter installation, which today gets pulled into the `site-packages` layer and should instead be part of the interpreter.
2. The single `site-packages` layer is far too coarse grained. Especially in the presence of heavy Python dependencies such as Torch, Numpy, Simplejpeg or Tensorflow, a single big `site-packages` layer fails to capitalize on the potential for layer/content reuse. Editing a single dependency invalidates the entire 3rdparty layer.
Making this better requires some amount of rethinking how `py_image_layer` is implemented, but I think there's some room here.
As noted by @thesayyn there are also some fairly tricky design constraints around trying to not just enable layer reuse for transit size but enable _action_ reuse.
Contributor guide
Research direction
Start by tracing how py_image_layer consumes the .runfiles tree from py_binary, including py_venv_* rules and the current interpreter, site-packages, and user-files split. Define a design that keeps interpreter-owned site-packages with the interpreter, creates finer-grained dependency layers, and accounts for both transit-size and action reuse; the issue names no files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100