apache / apache/buildstream

Cached failed action is used when retrying a failed build

Open
#2,020 0 comments 0 reactions 0 assignees View on GitHub
bug remote execution
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

When using remote execution, if a build fails the failure is saved in the action cache. However builds can fail for a variety of reasons (e.g. running out of resources) that are not necessarily reproducible.

Buildstream offers to retry at the command line and also offers the `--retry-failed` command line argument for this purpose. However, it will still get a cached failure from the action cache.

Here is an example log (I extracted just the things relevant to the element in question):
```
[11:08:35][--:--:--][8524a2f2][ pull:freedesktop-sdk.bst:components/fontconfig.bst] START freedesktop-sdk/components-fontconfig/8524a2f2-pull.20250611-110835.log
[11:08:35][--:--:--][8524a2f2][ pull:freedesktop-sdk.bst:components/fontconfig.bst] STATUS Pulling artifact 8524a2f2 <- https://gbm.gnome.org:11004
[11:08:35][--:--:--][8524a2f2][ pull:freedesktop-sdk.bst:components/fontconfig.bst] STATUS Pulling data for artifact 8524a2f2 <- https://gbm.gnome.org:11004
[11:08:35][--:--:--][8524a2f2][ pull:freedesktop-sdk.bst:components/fontconfig.bst] INFO Pulled artifact 8524a2f2 <- https://gbm.gnome.org:11004
[11:08:35][--:--:--][8524a2f2][ pull:freedesktop-sdk.bst:components/fontconfig.bst] INFO Discarded failed build
[11:08:35][00:00:00][8524a2f2][ pull:freedesktop-sdk.bst:components/fontconfig.bst] SKIPPED Pull
[11:08:35][--:--:--][8524a2f2][ fetch:freedesktop-sdk.bst:components/fontconfig.bst] START freedesktop-sdk/components-fontconfig/8524a2f2-fetch.20250611-110835.log
[11:08:35][--:--:--][8524a2f2][ fetch:freedesktop-sdk.bst:components/fontconfig.bst] STATUS Pulling source 031032b8 <- https://gbm.gnome.org:11004
[11:08:36][--:--:--][8524a2f2][ fetch:freedesktop-sdk.bst:components/fontconfig.bst] STATUS Pulling data for source 031032b8 <- https://gbm.gnome.org:11004
[11:08:36][--:--:--][8524a2f2][ fetch:freedesktop-sdk.bst:components/fontconfig.bst] INFO Pulled source 031032b8 <- https://gbm.gnome.org:11004
[11:08:36][00:00:00][8524a2f2][ fetch:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS freedesktop-sdk/components-fontconfig/8524a2f2-fetch.20250611-110835.log
[11:08:37][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] START freedesktop-sdk/components-fontconfig/8524a2f2-build.20250611-110837.log
[11:08:37][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] START Staging dependencies at: /
[11:09:04][00:00:26][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Staging dependencies at: /
[11:09:04][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] START Integrating sandbox
[11:09:32][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] INFO Action result found in action cache
[11:09:48][00:00:44][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Integrating sandbox
[11:09:48][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] START Staging sources
[11:09:48][00:00:00][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Staging sources
[11:09:48][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] START Running commands
[11:09:48][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] INFO Action result found in action cache
[11:09:49][00:00:00][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] FAILURE Running commands
[11:09:49][--:--:--][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] START Caching artifact
[11:09:49][00:00:00][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] SUCCESS Caching artifact
[11:09:49][00:01:11][8524a2f2][ build:freedesktop-sdk.bst:components/fontconfig.bst] FAILURE Command failed
```

You can see that bst pulled a cached failure and promptly discarded it, but it then proceeded to get the failed action from the action cache and not actually retry the build.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the --retry-failed command-line path and the action-cache lookup used during remote execution. Reproduce the logged scenario with a failed cached action; done means retrying bypasses the cached failure and runs the build action again.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.