Refactor --separate-weights to use named build contexts and Dockerfile.dockerignore

Problem

--separate-weights has three open bugs that share a root cause: the current implementation mutates the user's .dockerignore in-place and relies on a separate -weights Docker image that can go stale.

• #1323 — cog push --separate-weights fails with 404 because the local -weights image was pruned but the manifest cache (.cog/cache/weights_manifest.json) still thinks it exists. Docker tries to pull r8.im/…-weights from the registry, which doesn't exist.
• #2548 — --separate-weights permanently modifies .dockerignore. If the build fails between backup and restore, the user's file is left corrupted with cog-generated weight exclusions. Subsequent builds without the flag then silently exclude weight files.
• #1917 — Weight auto-detection makes wrong guesses, and because those guesses feed into .dockerignore mutation, a bad guess breaks the build rather than just affecting layer efficiency.

We'd like to make --separate-weights robust enough to enable by default.

Proposed approach

Replace the two-image build + .dockerignore mutation with a single Docker build using two BuildKit features that the codebase already has plumbing for:

Named build contexts

The generated Dockerfile becomes:

FROM weights
COPY checkpoints /src/checkpoints

FROM nvidia/cuda:11.8.0-...
# ... setup, pip install, etc ...
COPY --from=0 --link /src/checkpoints /src/checkpoints
COPY . /src

weights is a named build context (BuildContexts: {"weights": projectDir}) pointing at the project root. BuildKit resolves FROM weights to that directory instead of trying to pull a Docker image. The COPY commands in that stage selectively copy only the detected weight files. COPY --from=0 --link in the main stage puts them in independent layers, so code changes don't invalidate the weight cache.

The plumbing for named build contexts already exists at pkg/docker/buildkit.go:87-96 — it just isn't used by the standard generator today.

Dockerfile.dockerignore

BuildKit looks for <Dockerfile>.dockerignore in the Dockerfile directory. If found, it uses that instead of .dockerignore from the context root. Since cog already writes the Dockerfile to a temp directory, we write the generated ignore rules (user's original rules + weight exclusions) alongside it as Dockerfile.dockerignore. The user's .dockerignore is never touched.

Why this is robust enough to enable by default

No file mutation. The user's .dockerignore is never read, modified, backed up, or restored. There's no window where a failure can corrupt the project.

No phantom image dependency. There's no -weights image that can go missing after a docker prune, and no .cog/cache/weights_manifest.json that can go stale. BuildKit handles all caching internally via content-addressable layer hashes.

Idempotent. Running cog build with or without separate weights produces the same image contents. The only difference is layer structure. Switching between modes doesn't leave any state behind.

Graceful degradation of weight detection. If FindWeights() guesses wrong about which files are weights, the build still succeeds — files just end up in the wrong layer. With the old approach, a bad guess could corrupt .dockerignore and break the build entirely.

Single build is faster. BuildKit parallelises stages internally, so the weights stage and setup stages (apt, pip) run concurrently. The old approach serialised two full Docker builds with two context uploads.

Key changes

Fixes #1323. Fixes #1917. Fixes #2548.

Start with the named build-context plumbing in pkg/docker/buildkit.go:87-96, then read the generators in pkg/dockerfile/generator.go and pkg/dockerfile/standard_generator.go. Trace the current two-build flow in pkg/image/build.go and the ImageBuildOptions definition in pkg/docker/command/command.go, using pkg/dockerfile/standard_generator_test.go as the test entry point. Done means a single BuildKit build uses named contexts and Dockerfile.dockerignore without mutating .dockerignore or relying on the weights manifest cache.