devcontainers / devcontainers/features

Version resolution over unauthenticated git makes feature installs fail hard, with no fallback

Open
#1,726 1 comment 0 reactions 1 assignee Claimed by @v-Kaniska244 View on GitHub
Dominant language
Shell
Stars
1.5k
Forks
621
Avg merge
2d 11h
Merged PRs (30d)
4

Description

## Summary

Most features resolve "latest" by running `git ls-remote --tags` against github.com through a copy of
`find_version_from_git_tags`. GitHub rations unauthenticated git over HTTPS per source IP
(https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/ lists
"cloning repositories over HTTPS"). When the request is refused the helper ends up with an empty version
list and calls `exit 1`, so a rationed network becomes a failed build rather than a slower one.

This is not specific to one feature. The helper is copy pasted into 11 install scripts in 6 slightly
different variants, and every one of them exits 1 on an empty list.

## Scope

Features that resolve a version through github.com git:

```
docker-in-docker, docker-outside-of-docker, github-cli, git-lfs, go, kubectl-helm-minikube,
nix, node, php, powershell, python, rust
```

`oryx` and `ruby` additionally `git clone` from github.com during install, and `copilot-cli` resolves its
prerelease channel with a raw `git ls-remote`. That is 15 of the 28 features in `src/` depending on
traffic GitHub rations.

## What it looks like

docker-in-docker at defaults, resolving compose:

```
Finished installing docker / moby!
fatal: could not read Username for 'https://github.com': No such device or address
(!) Invalid compose_version value: latest\nValid values:\n
ERROR: Feature "Docker (Docker-in-Docker)" (ghcr.io/devcontainers/features/docker-in-docker) failed to install!
```

node, resolving nvm, in the same build:

```
fatal: could not read Username for 'https://github.com': No such device or address
Invalid NVM_VERSION value: latest
ERROR: Feature "Node.js (via nvm), yarn and pnpm." (ghcr.io/devcontainers/features/node) failed to install!
```

The credentials wording is misleading and costs a lot of debugging time. The refusal arrives in two
shapes. Sometimes git prints the server message, "GitHub is temporarily limiting some unauthenticated
downloads to protect the stability of the platform". Sometimes the answer is 401 with an auth challenge,
git tries to prompt, finds no tty in a build, and reports "could not read Username". Both are the same
limit, and neither is a proxy, a DNS fault or a missing credential.

## Measurements

From one container on an ordinary university network, base `ubuntu:24.04`, git 2.43.0.

Only part of the traffic is rationed. Protocol v2 splits `ls-remote` into `GET /info/refs` plus
`POST /git-upload-pack`, and the POST is the rationed half. Interleaved, same repo, same window:

```
protocol v2 (git default): ok=4 throttled=10
protocol v0 (legacy): ok=14 throttled=0
```

Clones are rationed harder, and the same content over plain HTTPS is not rationed at all:

```
git clone --depth 1 of a small repo : ok=2 failed=4
codeload.github.com tarball : ok=6 failed=0
```

The git endpoints return no rate limit headers, only `x-github-request-id`, so an install script cannot
detect the condition or back off. A REST reply by contrast carries `x-ratelimit-remaining` and a reset.

Because a build performs several of these lookups in a row, the failure rate compounds. A default
docker-in-docker plus node build makes four rationed requests, so at 95 percent per request the build
succeeds 81 percent of the time, and at 30 percent per request it succeeds under 1 percent of the time.

## Three defects

**1. A failed lookup is fatal.** The helper exits 1 as soon as the list is empty:

```bash
version_list="$(git ls-remote --tags ${repository} | grep -oP "${regex}" | ...)"
...
if [ -z "${!variable_name}" ] || ! echo "${version_list}" | grep "^${!variable_name//./\\.}$" ...; then
err "Invalid ${variable_name} value: ${requested_version}\nValid values:\n${version_list}"
exit 1
fi
```

Fallbacks that do exist sit downstream of this point. In docker-in-docker, `fallback_compose` and
`fallback_buildx` only cover a failed artifact download, so the earlier resolution failure never reaches
them.

**2. Pinning an exact version does not avoid the network.** The guard meant to skip the lookup is

```bash
if [ "$(echo "${requested_version}" | grep -o "." | wc -l)" != "2" ]; then
```

`grep -o "."` matches any character, so this counts characters rather than dots. Every realistic value,
`2.40.0` or `0.40.7` included, is longer than two characters and goes to the network. A two character
value skips the lookup but then fails the validation above, which compares against a list only the
network call can produce. There is no offline safe value except `none`.

**3. Some lookups are redundant.** docker-in-docker with `moby: true` installs `moby-compose` from apt
unconditionally, and the moby packages provide buildx too, then it resolves and downloads a second copy
from github.com. In an image built here, apt provided `moby-buildx 0.36.1` and `moby-compose 5.5.0`
alongside a feature written `/usr/local/bin/docker-compose`.

## Suggested fix

Fallback logic instead of `exit 1`. The repo already implements the mirror image of this in
`get_previous_version`, which calls `api.github.com`, detects `API rate limit exceeded` and then falls
back to GitHub tags. The same idea is missing in the other direction.

1. When `git ls-remote` yields nothing, try another source before failing. The REST API is one option, but
it is 60 requests per hour unauthenticated and shared per IP, so it belongs in a chain rather than as a
replacement single point of failure. Tarballs from `codeload.github.com` and the distro package are
unrationed alternatives.
2. Degrade to a known good pinned version as the last step, so a rationed network yields a slightly older
tool instead of a failed build.
3. Fix the character count so an explicitly pinned version skips the network entirely, and do not validate
a pinned value against a list that requires the network to exist.
4. Skip the resolution when the distro package already provides the tool.
5. Consider centralizing the helper. Eleven copies in six variants means any fix has to be repeated, and
they have already drifted apart.

## Repro

```jsonc
{
"image": "ubuntu:24.04",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:4.1.0": {},
"ghcr.io/devcontainers/features/node:2": {}
}
}
```

Build from a shared or busy egress IP. To force it deterministically, block git smart HTTP to github.com
while leaving `curl https://github.com` working.

## Workaround for anyone hitting this

In the base image, before the features run:

```dockerfile
RUN git config --system protocol.version 0
ENV METHOD=script
```

The first keeps every lookup on the half of the protocol that is not rationed. The second makes nvm's
installer fetch a tarball rather than clone, which node needs on top. Verified with a full
`devcontainer build --no-cache` of seven features including docker-in-docker at defaults and node.

## Versions

ghcr.io/devcontainers/features/docker-in-docker 4.1.0 and node 2.1.0 as the two examples above,
@devcontainers/cli 0.88.0, base ubuntu:24.04, git 2.43.0, podman 5.4.2 with buildah 1.39.3.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.