LoadWithParents is inefficient
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.3k
- Forks
- 1.5k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 48
Description
On importing remote cache cachemanager.LoadWithParents is called that internally calls worker.FromRemote. The reason for this call is that on cache match we don't just want to load matched record but save cache metadata for all the parent layers of that record as well. So that when the next build happens with only local cache and it gets a partial match it still works.
LoadWithParents currently will load the matched record, which means also loading its parent layers. And then does the same check for all the parent records, verifying each of their parent layers as well. That leads to n**2 complexity. The checks themselves are fast so this isn't very noticeable unless the image grows very big but looks like it is slower in the moby integration where verification of existing layers happens through downloadmanager and therefore does checks that access the local disk.
I didn't find a very clear solution on how to fix this. Changes to function signatures are needed and may need to pass a cache map. Logically, once a parent record has been loaded we only need to verify a layer on top of it. But keeping the state about the parent layer already being loaded is tricky.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading cachemanager.LoadWithParents and the worker.FromRemote call described in the issue. Trace how matched records and parent layers are loaded, including the moby integration path through downloadmanager. Done means parent verification no longer repeats the same work for every ancestor while preserving cache metadata behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- build-system, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100