replicate / replicate/cog

Fail while pushing with --separate-weights

Open
#1,323 17 comments 14 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

go next type/bug
Dominant language
Go
Stars
9.5k
Forks
696
Avg merge
7d 19h
Merged PRs (30d)
2

Description

Today I started to face the following issue, while using cog push --separate-weights
Although I was able to push the model without the flag --separate-weights

On the error below, I just replaced the project and model name by and <model-name, respectively.

$ cog push --separate-weights
⚠ Cog doesn't know if CUDA 11.2.2 is compatible with PyTorch 1.13.1. This might cause CUDA problems.
Building Docker image from environment in cog.yaml as r8.im/<project-name>/<model-name> ...
Weights unchanged, skip rebuilding and use cached image...
[+] Building 4.0s (7/7) FINISHED                                                      docker:default
 => [internal] load .dockerignore                                                               0.0s
 => => transferring context: 22.25kB                                                            0.0s
 => [internal] load build definition from Dockerfile                                            0.0s
 => => transferring dockerfile: 4.41kB                                                          0.0s
 => resolve image config for docker.io/docker/dockerfile:1.4                                    1.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a1  0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04          1.2s
 => ERROR [internal] load metadata for r8.im/<project-name>/<model-name>  2.2s
 => [auth] <project-name>/<model-name> -weights:pull token for r8.im    0.0s
------
 > [internal] load metadata for r8.im/<project-name>/<model-name>-weights:latest:
------
Dockerfile:2
--------------------
   1 |     #syntax=docker/dockerfile:1.4
   2 | >>> FROM r8.im/<project-name>/<model-name>-weights AS weights
   3 |     FROM nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04
   4 |     ENV DEBIAN_FRONTEND=noninteractive
--------------------
ERROR: failed to solve: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://r8.im/_token?scope=repository%3A<project-name>%2F<model-name>-weights%3Apull&service=us-docker.pkg.dev: 404 Not Found
ⅹ Failed to build runner Docker image: Failed to build Docker image: exit status 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cog push --separate-weights path and reproduce the Docker build failure shown in the issue. Inspect Dockerfile line 2, where the -weights image is used, and the registry token request that returns 404. Done means pushing with --separate-weights completes without the failed metadata lookup or authorization error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.