bazel-contrib / bazel-contrib/rules_oci

FR: Shallow pull for oci_pull

Open
#497 3 comments 3 reactions 0 assignees View on GitHub
enhancement need: discussion performance
Dominant language
Starlark
Stars
423
Forks
213
Avg merge
53m
Merged PRs (30d)
1

Description

Currently, base image is input to oci_image. This has some downsides

- Pushing giant blobs to remote action cache is slow and takes more time than building the target from scratch
- Copying blobs from base image into the oci_image output takes time (see #365)
- Downloading the base image from the internet takes time

I am proposing a shallow pull feature that will prevent this from happening by only downloading necessary image metadata such as `manifest` and `config` so we can mutate the config and the manifest. We never mutate existing layers so this is safe.

However, it's not as easy it sounds since we have to

- instruct `crane` to be happy with missing blobs on disk (to be done)
- oci_tarball has to be aware of shallow oci-layouts and create necessary metadata on its tarball `manifest` so the runtime knows where to get the blobs from
- oci_push needs to know how to deal oci-layouts, eg knowing how to `mount` blobs (see https://github.com/google/go-containerregistry/issues/1321). Feature is already there but we might need to change oci_push a little bit for this to work.

We won't be offering this feature by default as we can't be certain every tooling will know how to deal with shallow oci-layouts.

Contributor guide

Open the contributing guide

Research direction

Start with the oci_pull, oci_tarball, and oci_push entry points named in the issue, then investigate how crane handles missing blobs. Check the linked go-containerregistry issue for blob mounting context. Done means shallow metadata-only pulls work through tarball creation and pushing without requiring existing layers locally.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.