aws / aws/deep-learning-containers

telemetry: _retrieve_os() returns empty string on Amazon Linux 2023 — 35 of 43 released images report no OS

Open
#6,529 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
559
Avg merge
1d 7h
Merged PRs (30d)
102

Description

## Summary

`deep_learning_container.py:_retrieve_os()` cannot parse Amazon Linux's `/etc/os-release`. It returns `""` on every Amazon Linux 2023 image, so the OS segment of the `aws-dlc-autogenerated-tag-do-not-delete` EC2 instance tag — and the OS dimension of DLC usage telemetry — is blank for most released DLCs.

## Root cause

The function matches `/etc/os-release` with two regexes:

```python
if re.match(r"^ID=\w+$", line):
name = re.search(r"^ID=(\w+)$", line).group(1)
if re.match(r'^VERSION_ID="\d+\.\d+"$', line):
version = re.search(r'^VERSION_ID="(\d+\.\d+)"$', line).group(1)
```

They assume an **unquoted** `ID` and a **quoted, dotted** `VERSION_ID`. That describes Ubuntu, not Amazon Linux.

Verbatim `/etc/os-release` from `amazonlinux:2023` (the base under `nvidia/cuda:*-amzn2023`, which the DLC Dockerfiles build `FROM`):

```
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
```

- `ID="amzn"` is quoted → `^ID=\w+$` does not match → `name` stays `""`
- `VERSION_ID="2023"` has no dot → `^VERSION_ID="\d+\.\d+"$` does not match → `version` stays `""`

Both captures fail independently, so the function returns `""`.

For contrast, `ubuntu:24.04` ships `ID=ubuntu` (bare) and `VERSION_ID="24.04"` (dotted) — both regexes match, which is why this was never caught.

## Observed effect

Running the current `_retrieve_os()` against those two real files:

```
_retrieve_os() on amzn2023 -> ''
_retrieve_os() on ubuntu24.04 -> 'ubuntu24.04'
```

The tag built in `tag_instance()` is `f"{framework}_{container_type}_{framework_version}_python{py_version}_{device}{cuda_version}_{os_version}"`, so on an AL2023 image it ends in a bare trailing underscore:

```
vllm_server_training_0.26.0_python3.12.0_gpu_cuda13.0_
```

instead of

```
vllm_server_training_0.26.0_python3.12.0_gpu_cuda13.0_amzn2023
```

## Measured impact

Across the released image configs on `main` (`release.release: true` with a `prod_image`), cross-referencing each config's `build.dockerfile` for whether it wires in `deep_learning_container.py` / `bash_telemetry`:

| | count |
|---|---|
| Unique released images | 46 |
| ...that ship the telemetry script | 43 |
| ...of those, running on `amzn2023` → **empty OS field** | **35 (81%)** |
| ...of those, running on Ubuntu → correct OS field | 8 |

Affected frameworks: `base`, `huggingface-pytorch-training`, `openfold3`, `pytorch_runtime`, `ray`, `sglang_server`, `sklearn`, `tensorflow`, `vllm_omni`, `vllm_server`, `xgboost`.

The share is growing, not shrinking — the repo is actively migrating images to Amazon Linux 2023, and telemetry was just extended to the base images in #6499.

## The repo already has the correct idiom

`scripts/ci/autocurrency/detect-versions.sh:111` reads the same file the right way:

```bash
bash -c 'source /etc/os-release && echo "${ID}${VERSION_ID}"'
```

That yields `amzn2023` / `ubuntu24.04` — exactly the `os_version` spelling used in `.github/config/image`. The telemetry script is the outlier.

## Reproduction

No container required:

```python
import sys
from unittest.mock import patch
sys.path.insert(0, "scripts/docker/telemetry")
import deep_learning_container as dlc
import io

AMZN = 'NAME="Amazon Linux"\nID="amzn"\nID_LIKE="fedora"\nVERSION_ID="2023"\n'
real = open
with patch("builtins.open", lambda f, *a, **k: io.StringIO(AMZN) if f == "/etc/os-release" else real(f, *a, **k)):
print(repr(dlc._retrieve_os()))
```

```
''
```

## Suggested fix

Parse `os-release` as the `key=value` format it is — split on the first `=`, strip surrounding quotes, take `ID` and `VERSION_ID` verbatim. That is a strict superset of the current regexes, so Ubuntu output is unchanged while Amazon Linux starts reporting `amzn2023`.

Worth noting separately: `_retrieve_os()` currently calls `open()` with no error handling, and `tag_instance()` calls it *before* its `try` block — so an unreadable `/etc/os-release` raises and kills the tagging process before it can tag at all.

PR: #6530

Contributor guide

Open the contributing guide

Research direction

Start in scripts/docker/telemetry/deep_learning_container.py and run the supplied Python reproduction with the Amazon Linux /etc/os-release sample. Done means _retrieve_os() reports amzn2023 for Amazon Linux while preserving the existing Ubuntu result; PR #6530 indicates work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.