gitpod-io / gitpod-io/leeway

Early upload successfully built artifacts to remote cache

Open
#74 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
183
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Bug description

Leeway, when building the 'installer:docker' target in a gitpod.io workspace has a lot of build dependencies..
The default concurrency limit is 16, so it starts building many pkgs in parallel..

In my case, there are always some temporary build errors with some pkgs that stops the entire build process.
The key problem is that Leeway only starts uploading build artifacts once at the end of a successful build (or missing to do so due to premature exit on failures ?):
☁️ uploading build artifacts to remote cache

Since due to the build failure, many packages that are previously built are not present in the remote (and local) cache, so when I restart the build, I see many of those are getting re-built from scratch again and again. This is causing almost an endless loop, if the temporary errors keep popping up & I keep re-running the build command.

The only workaround I found was to specify lighter build targets from the list of components that are not present in the remote cache.. let them finish and see they are getting uploaded.. I can typically get the whole build done in like 10-15 total attempts, which is super annoying & time consuming.

Can this be improved, like by triggering both an image push & remote cache artifact upload at the end of each successfully built component, when it prints package build succeded (has a typo)?

Another idea for improvement: actually, uploading the artifacts to the remote cache at the very end of the build process is fine, but Leeway should wait for all forked background / parallel builds' results before exiting and still upload the successfully built targets' artifacts to remote cache in case of an overall build failure.. So let's say the entire build includes 25 targets, there is a build error with a single target (may fail the dependent ones), I would expect Leeway to print a stat in the end of how many of the 25 were built successfully, explicitly mentioning the failed targets, as well as that it uploads the successful ones to the remote cache, so on next attempt those pkgs will not get rebuilt.

Steps to reproduce

In description. I can provide build logs, if needed. Just let me know.

Expected behavior

No response

Example repository

No response

Anything else?

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the installer:docker build with parallel targets and observe the package build succeeded and remote-cache upload messages. Trace the build completion and failure handling, then verify that successful targets are uploaded and reported even when another target fails, so a retry does not rebuild them.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.