aspect-build / aspect-build/rules_py

[FR]: Container image layers per-package

Open
#244 8 comments 1 reaction 0 assignees View on GitHub
feature request
Dominant language
Starlark
Stars
145
Forks
97
Avg merge
1d 1h
Merged PRs (30d)
71

Description

### What is the current behavior?

Currently our best-effort to make a layered image for Python is three layers: interpreter, site-packages, app:
https://github.com/aspect-build/bazel-examples/blob/main/oci_python_image/py_layer.bzl

However, when a package appears under multiple applications, the site-packages layer for each app will have a duplicate of that package, making the build outputs larger. When pytorch is one such package (it is many gigabytes for the GPU variant) then this problem leads to builds taking minutes just to fetch build inputs from a remote cache.

### Describe the feature

We can probably do something with aspects to visit each third-party package and produce a `tar` file containing its site-packages entry as a distinct layer. Then when we build a container image for an application these `tar` files should just get re-used rather than being re-built.

Contributor guide

Open the contributing guide

Research direction

Start by reading oci_python_image/py_layer.bzl and trace how the current interpreter, site-packages, and app layers are assembled. Investigate how aspects could visit third-party packages and produce per-package tar layers. Done means applications reuse those package layers instead of rebuilding duplicate site-packages contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
build-system, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.