moby / moby/buildkit

LoadWithParents is inefficient

Open
#2,184 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

On importing remote cache cachemanager.LoadWithParents is called that internally calls worker.FromRemote. The reason for this call is that on cache match we don't just want to load matched record but save cache metadata for all the parent layers of that record as well. So that when the next build happens with only local cache and it gets a partial match it still works.

LoadWithParents currently will load the matched record, which means also loading its parent layers. And then does the same check for all the parent records, verifying each of their parent layers as well. That leads to n**2 complexity. The checks themselves are fast so this isn't very noticeable unless the image grows very big but looks like it is slower in the moby integration where verification of existing layers happens through downloadmanager and therefore does checks that access the local disk.

I didn't find a very clear solution on how to fix this. Changes to function signatures are needed and may need to pass a cache map. Logically, once a parent record has been loaded we only need to verify a layer on top of it. But keeping the state about the parent layer already being loaded is tricky.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading cachemanager.LoadWithParents and the worker.FromRemote call described in the issue. Trace how matched records and parent layers are loaded, including the moby integration path through downloadmanager. Done means parent verification no longer repeats the same work for every ancestor while preserving cache metadata behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
build-system, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.