LukeMathWalker / LukeMathWalker/cargo-chef

`cargo-chef` doesn't cache local dependencies

Open
#75 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
2.7k
Forks
146
PR merge metrics
No merged PRs in 30d

Description

(The repo this issue is based on is private. I will describe the issue first, then try to recreate in a public minimal repo for further clarity and reproducibility.)

I have a project directory with structure
```
Cargo.toml # [virtual manifest](https://doc.rust-lang.org/cargo/reference/workspaces.html#virtual-manifest) cargo.toml
src
- libA
- libB
- libC
- serviceZ
- serviceY
- serviceX
- ...
...
```

Each of the services has individual `Cargo.toml` files and its own `Dockerfile`.

I've tried adding `cargo-chef` to one of these services (currently, we are either not being cache friendly or using a `cargo new` approach), say, `serviceZ`. `cargo-chef` does excellently at caching external dependencies for the entire workspace, but because it skelefies `libA`, `libB`, and `libC`, it has to rebuild these local library dependencies for each change in `serviceZ`. (The actual repo is too large to individually specify what dependencies to copy over and doing so would lead to keeping dependencies between the project and the `Cargo.toml` file in sync.) These end up consuming most of the compilation time: using cargo chef can bring subsequent runs from `404s` down to `270s`, but using a `cargo new` approach brings subsequent runs down from `404s` to `70s`. I can also visually see that we are recompiling local dependencies when running `cargo build` in the `cargo chef` approach while only `serviceZ` is compiled in the `cargo new` approach.

This is related to https://github.com/LukeMathWalker/cargo-chef/issues/74 and https://github.com/LukeMathWalker/cargo-chef/issues/64 in that I want very fine control over what I skelefy.

A quick aside: the `cargo new` approach suffers from changes in `serviceZ` invalidating the entire `COPY src src` line, leading to discarding the dependency cache on source changes. The way I've gotten around it is to separate this process into several stages, similar to the approach in `cargo chef`, but instead of sharing the `recipe.json` file between stages, I'm sharing an entire directory that should be the same between runs:
```dockerfile
FROM rust as assembler
WORKDIR /usr/app
COPY src src
RUN rm -rf src/serviceZ
RUN USER=root cargo new --bin ./src/serviceZ
COPY src/serviceZ/Cargo.toml src/serviceZ/
# because the above stage should be the same between changes to `serviceZ`,
# we can cache copying it to other stages

FROM rust as cacher
WORKDIR /usr/app
COPY Cargo.lock Cargo.lock
COPY rust-patches rust-patches
COPY vendored vendored
COPY .cargo .cargo
COPY Cargo.toml Cargo.toml
COPY --from=assembler /usr/app/src ./src
RUN cargo build --offline -p serviceZ && rm target/debug/deps/serviceZ*

FROM rust as builder
# actually copies over contents of serviceZ and builds it using cached items in cacher stage
...
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the per-service Dockerfile, the workspace Cargo.toml, and the cargo-chef workflow described in the issue; compare it with the staged cargo new approach. Reproduce the workspace build if possible, then verify that changing serviceZ does not rebuild unchanged local libraries while external dependencies remain cached.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.