AlexsLemonade / AlexsLemonade/refinebio

Docker cache miss during deploy

Open
#3,340 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
135
Forks
21
PR merge metrics
No merged PRs in 30d

Description

### Context

During deploy process _some_ docker images still being rebuilt from scratch.

### Problem or idea

Smasher:

```
#9 importing cache manifest from ccdlstaging/dr_smasher_cache:latest
#9 DONE 0.3s

#6 [ 1/12] FROM docker.io/ccdlstaging/dr_base:v1.45.7-dev@sha256:776a8c605a6a04914572f59d0be4989b8c9fe05011d62466bd167af90868c2f3
#6 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 1.56MB 0.1s done
#5 DONE 0.1s

#10 [ 2/12] WORKDIR /home/user
#10 CACHED

#11 [ 3/12] COPY workers/R/dependencies/smasher/renv.lock .
#11 DONE 0.1s

#12 [ 4/12] COPY common/R/renv_load.R .
#12 DONE 0.0s

#13 [ 5/12] RUN Rscript renv_load.R
#13 0.463 Installing package into ‘/usr/local/lib/R/site-library’
#13 0.463 (as ‘lib’ is unspecified)
#13 2.414 trying URL 'https://bioconductor.org/packages/3.6/bioc/src/contrib/BiocInstaller_1.28.0.tar.gz'
#13 2.709 Content type 'application/octet-stream' length 18624 bytes (18 KB)
#13 2.722 ==================================================
```

Salmon:

```
#9 importing cache manifest from ccdlstaging/dr_salmon_cache:latest
#9 DONE 0.2s

#6 [ 1/14] FROM docker.io/ccdlstaging/dr_base:v1.45.7-dev@sha256:776a8c605a6a04914572f59d0be4989b8c9fe05011d62466bd167af90868c2f3
#6 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 14.06kB 0.1s done
#5 DONE 0.1s

#10 [ 2/14] WORKDIR /home/user
#10 CACHED

#11 [ 3/14] COPY workers/R/dependencies/salmon/renv.lock .
#11 CACHED

#12 [ 4/14] COPY common/R/renv_load.R .
#12 DONE 0.1s

#13 [ 5/14] RUN Rscript renv_load.R
#13 0.358 Installing package into ‘/usr/local/lib/R/site-library’
#13 0.358 (as ‘lib’ is unspecified)
#13 2.507 trying URL 'https://bioconductor.org/packages/3.6/bioc/src/contrib/BiocInstaller_1.28.0.tar.gz'
#13 2.803 Content type 'application/octet-stream' length 18624 bytes (18 KB)
#13 2.821 ==================================================
```
### Solution or next step

Figure out the layer cache invalidation reason and fix it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the Docker build logs for Smasher and Salmon, especially the cache manifest import and the COPY steps for workers/R/dependencies/*/renv.lock and common/R/renv_load.R. Trace why the RUN Rscript renv_load.R layer is rebuilt during deploy. Done means the layer cache is reused when inputs have not changed and the affected images no longer rebuild dependencies from scratch.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, r
Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.