microsoft / microsoft/AzureTRE
Reexamine Docker Hub image pulls and leverage ACR as cache
Nobody has claimed this yet.
- #4918 by @copilot-swe-agent — closed without merging
- Dominant language
- Python
- Stars
- 235
- Forks
- 192
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
Is your feature request related to a problem? Please describe.
Pipeline runs fail with a 429: TOOMANYREQUESTS error whenever we hit rate limits on Docker Hub. I've seen this happen several times over the past week, blocking pipeline runs for 6 hours. This blocks all development and CICD pipeline runs until that limit expires. Here's an example.
We need to rethink how often we pull images from Docker Hub into ACR and whether we do a preflight check to ensure the image is already in ACR (and corresponding SHA matches). At present it's only the Nexus image that I can see but I would expect a good solution to provide a standard procedure for any public image we pull.
At present the flow is:
- GitHub Actions publishes shared bundles. The Nexus bundle is included in
.github/workflows/deploy_tre_reusable.yml at templates/shared_services/sonatype-nexus-vm/, and the job runs:make bundle-build bundle-publish DIR=${{ matrix.BUNDLE_DIR }} make bundle-buildcallsdevops/scripts/bundle_runtime_image_build.sh:8.- The Nexus Porter file declares an imported runtime image:
templates/shared_services/sonatype-nexus-vm/porter.yaml:9
custom:
runtime_image:
name: sonatype/nexus3
import:
source: docker.io/sonatype/nexus3
tag: "3.77.2"
- Because that import block exists,
bundle_runtime_image_build.shruns:
devops/scripts/bundle_runtime_image_build.sh:14
az acr import --name "${ACR_NAME}" \
--source "${source_image}:${version}" \
--image "${image_name}:${version}" \
--force
At deploy time, Terraform creates an Ubuntu VM. The VM uses a normal Canonical marketplace image in templates/shared_services/sonatype-nexus-vm/terraform/vm.tf:123. Cloud-init then runs /shared_services/sonatype-nexus-vm/scripts/deploy_nexus_container.sh:7, which pulls: ${ACR_NAME}.azurecr.io/sonatype/nexus3:${NEXUS_IMAGE_TAG}.
Describe the solution you'd like
For our pipelines, implement a form of a cache hit mechanism where we check ACR prior to pulling images from Docker Hub which already exist in ACR. This would reduce the number of pulled images to those which are needed at build time.
At present, if the container image import worked, the VM pulls from ACR. It only falls back to Docker Hub if ACR_NAME is empty. Before az acr import is run in devops/scripts/bundle_runtime_image_build.sh:14, we should add a check to see whether the relevant Sonatype Nexus image exists (e.g. sonatype/nexus3:3.77.2) in the target ACR, then skip the import if the image is present:
if az acr repository show-tags \
--name "${ACR_NAME}" \
--repository "${image_name}" \
--query "[?@=='${version}'] | [0]" \
--output tsv | grep -qx "${version}"; then
echo "Image ${image_name}:${version} already exists in ACR ${ACR_NAME}; skipping import"
exit 0
fi
That would avoid hitting Docker Hub on every publish when the image is already cached in ACR.
Another thing to consider is in templates/shared_services/sonatype-nexus-vm/scripts/deploy_nexus_container.sh:48: after ACR login but before docker pull, run docker manifest inspect "$NEXUS_IMAGE" and fail with a clear “missing from ACR” message.
Describe alternatives you've considered
I haven't considered any alternatives other than the above just yet, happy to be challenged.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.