LukeMathWalker / LukeMathWalker/cargo-chef

Add `--external-only` flag to `cargo chef prepare` to split third-party and workspace-internal cache layers

Open
#359 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.7k
Forks
146
PR merge metrics
No merged PRs in 30d

Description

## Problem

In large Cargo workspaces the standard `cargo-chef` pattern produces a single cook layer that is invalidated by **any** structural change to the workspace manifests — not just changes to external dependencies.

```dockerfile
RUN cargo chef prepare --recipe-path recipe.json # captures all manifests
RUN cargo chef cook --recipe-path recipe.json # invalidated too often
```

`recipe.json` includes every `[dependencies.my-crate] path = "../my-crate"` entry from every workspace manifest. Any of the following changes forces a full recompile of all third-party crates:

| Change | Invalidates cook layer? |
|---|---|
| Adding a new workspace-internal crate | **Yes — unexpected** |
| Splitting one internal crate into two | **Yes — unexpected** |
| Adding a `[[bin]]` target to a workspace crate | **Yes — unexpected** |
| Renaming a workspace crate | **Yes — unexpected** |
| Adding a new external dep | Yes — expected |

In the real-world project that motivated this issue ([torrust-tracker](https://github.com/torrust/torrust-tracker), 26 workspace crates) almost every feature branch touches at least one manifest. The cook cache is effectively **always cold**, and all third-party crates are recompiled from source on every CI run.

This has been reported before: see #314 and #75.

## Proposed solution

Add an `--external-only` flag to `cargo chef prepare` that strips all intra-workspace `path = "..."` dependencies before serialising the recipe, producing a recipe that is **only** invalidated when an external dependency changes.

```dockerfile
# Stable layer — only invalidated when external deps change
RUN cargo chef prepare --external-only --recipe-path recipe-thirdparty.json
RUN cargo chef cook --release --recipe-path recipe-thirdparty.json

# Fast layer — invalidated on any manifest change, but third-party is already warm
RUN cargo chef prepare --recipe-path recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
```

In a 3-crate example workspace the savings are **83×**: cold cook goes from 5.8 s → 0.07 s once the thirdparty layer is warm.

## Implementation

I have a working implementation ready:

- New `src/skeleton/external_only.rs` module using `toml::Value` (already a dependency) for structurally-correct stripping — not regex
- Handles: top-level dep sections, `[target.'cfg(...)'.dependencies]`, `[workspace.dependencies]`, and `{ workspace = true }` references to path deps (two-pass approach)
- Removes local (no-`source`) packages from the lock file so workspace-membership changes don't bust the recipe
- 6 unit tests + 3 integration tests
- An `examples/workspace-split-cache/` directory that demonstrates the problem and both solutions (Python workaround + native flag), with a step-by-step README

I will open a PR shortly.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the proposed src/skeleton/external_only.rs module and the examples/workspace-split-cache/README.md, then inspect the six unit and three integration tests described in the issue. Done means --external-only produces a recipe without intra-workspace path dependencies or local lockfile packages, while normal prepare behavior remains unchanged and the tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
build-system, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.