Improve remote cache hit ratio

Hello,

Here is how works the current remote cache system (I hope I'm not saying crap :) )

• The user sepecify some tag to use as cache with --cache-from
• Buildkit download manifests accordingly.
• Builkdit download layer if one is needed.

My standard config is to use cache from latest_<master> and latest_<current_branch_name>. The issue is, if my branch is not sync with master, I cannot leverage cache of older master build. To extend this, I'm not able to use the cache of other build, not because they are not cached, but because I do not know which tag I should use in --cache-from.

That's why I would like to consider to add a new API between buildkit and remote cache: do you have any manifest which contains the layer xxx? This will allow to use the cached layer of any other build, and not worry about the tags to put in --cache-from. And this is what is already done with local cache (if I'm not wrong).

If you agree, I will be happy to work on this. I'm able to do the remote cache part (in the s3 one :) ), but I need guidance / help to add the API call inside the buildkit reactor.

Thx

Start by reading the remote cache implementation, especially the S3 backend, and trace the BuildKit reactor entry point that handles cache lookups. The issue's proposed outcome is an API that can find a manifest containing a requested layer without requiring users to know cache tags; confirm the design and integration points with maintainers before coding.