aspect-build / aspect-build/aspect-cli
delivery: recover when the local action cache outlives the remote cache
- Dominant language
- Starlark
- Stars
- 165
- Forks
- 45
- Avg merge
- 1d 47m
- Merged PRs (30d)
- 35
Description
## Problem
Selective `aspect delivery` can leave unchanged targets pending when the normal build reuses a persistent local Bazel action-cache entry whose corresponding remote-cache entry has expired or been evicted.
The phase-one build succeeds, but the checksum phase runs on the separate `-checksum` output base with `--experimental_remote_require_cached`. If an upstream action is no longer present remotely, Bazel never reaches the target's `DeliveryHash` action, so the gRPC log contains no output SHA for that target.
This turns remote-cache retention into a correctness requirement for selective delivery even though the artifact is still valid in the runner's local action cache.
## Observed behavior
Environment:
- Aspect CLI `v2026.28.4`
- Bazel `9.2.0`
- persistent Aspect Workflows runner output base
- writable BuildBarn remote cache
An initial run built and delivered a set of image targets successfully. A later run used the same CLI version and checksum architecture, but phase one reused the warm local action cache. Phase two could resolve most delivery digests, while targets whose upstream actions were absent from the remote cache remained pending.
Representative upstream failures were `LayerTar` and `LayerInputFilesCAS` actions reporting:
```text
Action must be cached due to --experimental_remote_require_cached but it is not
```
The final summary reported the affected targets as pending rather than skipped, despite no source changes to those targets.
```text
0 delivered, skipped, pending, 0 failed
```
Rolling the release build back onto the shared output base did not help because the checksum phase correctly remains on its dedicated `-checksum` server.
## Expected behavior
Selective delivery should recover from an ordinary remote-cache miss. Rebuilding an artifact needed to calculate its digest is acceptable, but an unchanged target must still be compared with delivery state and skipped. Recovery must not dispatch or `bazel run` every deliverable.
## Proposed fix
Add one bounded repair attempt inside `_get_output_shas` after parsing the first checksum probe:
1. Collect targets absent from `output_shas`.
2. Run `bazel build` for only those unresolved labels, using the same unstamped flags/configuration as phase one plus:
- `--nouse_action_cache` to prevent the warm local action cache from short-circuiting remote lookup/upload
- `--remote_upload_local_results`
- `--noremote_cache_async` so repaired results are visible before retrying
3. Re-run the cache-only checksum probe for only those labels.
4. Merge recovered output SHAs into the original result.
5. Preserve the current warning/error behavior for labels still unresolved after the single retry.
This is a build-only repair. Phase-three state comparison and dispatch remain unchanged, so containers are pushed only when the recovered digest differs from recorded delivery state.
`cache_diff.axl` already uses `--nouse_action_cache` to force fresh remote lookups instead of trusting the persistent local action cache:
https://github.com/aspect-build/aspect-cli/blob/v2026.28.4/crates/aspect-cli/src/builtins/aspect/cache_diff.axl#L104-L183
The relevant Delivery implementation is here:
https://github.com/aspect-build/aspect-cli/blob/v2026.28.4/crates/aspect-cli/src/builtins/aspect/delivery.axl#L383-L710
## Suggested regression test
1. Run selective Delivery once to populate delivery state, the local output base, and the remote cache.
2. Preserve the local output base but clear or replace only the remote action cache.
3. Run selective Delivery again.
4. Assert that the repair build occurs for unresolved labels.
5. Assert that the target is reported `SKIP`, not `pending` or `delivered`, and that its entrypoint is not dispatched a second time.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with _get_output_shas in crates/aspect-cli/src/builtins/aspect/delivery.axl and compare its cache handling with cache_diff.axl lines 104-183. Reproduce the two-run scenario while preserving the local output base and clearing the remote action cache. Done means one bounded repair build recovers unresolved SHAs, the unchanged target is reported SKIP, and its entrypoint is not dispatched again.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100