aspect-build / aspect-build/rules_py

[FR]: More granular `py_image_layer` behavior

Open
#555 1 comment 1 reaction 0 assignees View on GitHub
need: discussion
Dominant language
Starlark
Stars
145
Forks
97
Avg merge
1d 1h
Merged PRs (30d)
71

Description

Today `py_image_layer` takes the `.runfiles` tree from a `py_binary` including the new `py_venv_*` rules and by default attempts to break that one input filegroup into three layers.

1. The "interpreter", assuming that the user is leaning on hermetic interpreters from Astral.
2. The `site-packages` tree(s), which allows for layer reuse between images which share the same _total set_ of 3rdparty input files.
3. User files / everything else.

While this makes sense as a basic splitting strategy, it leaves much to be desired.

1. The interpreter has its own `site-packages` files which _are_ part of the interpreter installation, which today gets pulled into the `site-packages` layer and should instead be part of the interpreter.
2. The single `site-packages` layer is far too coarse grained. Especially in the presence of heavy Python dependencies such as Torch, Numpy, Simplejpeg or Tensorflow, a single big `site-packages` layer fails to capitalize on the potential for layer/content reuse. Editing a single dependency invalidates the entire 3rdparty layer.

Making this better requires some amount of rethinking how `py_image_layer` is implemented, but I think there's some room here.

As noted by @thesayyn there are also some fairly tricky design constraints around trying to not just enable layer reuse for transit size but enable _action_ reuse.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how py_image_layer consumes the .runfiles tree from py_binary, including py_venv_* rules and the current interpreter, site-packages, and user-files split. Define a design that keeps interpreter-owned site-packages with the interpreter, creates finer-grained dependency layers, and accounts for both transit-size and action reuse; the issue names no files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.