aws / aws/deep-learning-containers
telemetry: _retrieve_os() returns empty string on Amazon Linux 2023 — 35 of 43 released images report no OS
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 559
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 102
Description
## Summary
`deep_learning_container.py:_retrieve_os()` cannot parse Amazon Linux's `/etc/os-release`. It returns `""` on every Amazon Linux 2023 image, so the OS segment of the `aws-dlc-autogenerated-tag-do-not-delete` EC2 instance tag — and the OS dimension of DLC usage telemetry — is blank for most released DLCs.
## Root cause
The function matches `/etc/os-release` with two regexes:
```python
if re.match(r"^ID=\w+$", line):
name = re.search(r"^ID=(\w+)$", line).group(1)
if re.match(r'^VERSION_ID="\d+\.\d+"$', line):
version = re.search(r'^VERSION_ID="(\d+\.\d+)"$', line).group(1)
```
They assume an **unquoted** `ID` and a **quoted, dotted** `VERSION_ID`. That describes Ubuntu, not Amazon Linux.
Verbatim `/etc/os-release` from `amazonlinux:2023` (the base under `nvidia/cuda:*-amzn2023`, which the DLC Dockerfiles build `FROM`):
```
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
```
- `ID="amzn"` is quoted → `^ID=\w+$` does not match → `name` stays `""`
- `VERSION_ID="2023"` has no dot → `^VERSION_ID="\d+\.\d+"$` does not match → `version` stays `""`
Both captures fail independently, so the function returns `""`.
For contrast, `ubuntu:24.04` ships `ID=ubuntu` (bare) and `VERSION_ID="24.04"` (dotted) — both regexes match, which is why this was never caught.
## Observed effect
Running the current `_retrieve_os()` against those two real files:
```
_retrieve_os() on amzn2023 -> ''
_retrieve_os() on ubuntu24.04 -> 'ubuntu24.04'
```
The tag built in `tag_instance()` is `f"{framework}_{container_type}_{framework_version}_python{py_version}_{device}{cuda_version}_{os_version}"`, so on an AL2023 image it ends in a bare trailing underscore:
```
vllm_server_training_0.26.0_python3.12.0_gpu_cuda13.0_
```
instead of
```
vllm_server_training_0.26.0_python3.12.0_gpu_cuda13.0_amzn2023
```
## Measured impact
Across the released image configs on `main` (`release.release: true` with a `prod_image`), cross-referencing each config's `build.dockerfile` for whether it wires in `deep_learning_container.py` / `bash_telemetry`:
| | count |
|---|---|
| Unique released images | 46 |
| ...that ship the telemetry script | 43 |
| ...of those, running on `amzn2023` → **empty OS field** | **35 (81%)** |
| ...of those, running on Ubuntu → correct OS field | 8 |
Affected frameworks: `base`, `huggingface-pytorch-training`, `openfold3`, `pytorch_runtime`, `ray`, `sglang_server`, `sklearn`, `tensorflow`, `vllm_omni`, `vllm_server`, `xgboost`.
The share is growing, not shrinking — the repo is actively migrating images to Amazon Linux 2023, and telemetry was just extended to the base images in #6499.
## The repo already has the correct idiom
`scripts/ci/autocurrency/detect-versions.sh:111` reads the same file the right way:
```bash
bash -c 'source /etc/os-release && echo "${ID}${VERSION_ID}"'
```
That yields `amzn2023` / `ubuntu24.04` — exactly the `os_version` spelling used in `.github/config/image`. The telemetry script is the outlier.
## Reproduction
No container required:
```python
import sys
from unittest.mock import patch
sys.path.insert(0, "scripts/docker/telemetry")
import deep_learning_container as dlc
import io
AMZN = 'NAME="Amazon Linux"\nID="amzn"\nID_LIKE="fedora"\nVERSION_ID="2023"\n'
real = open
with patch("builtins.open", lambda f, *a, **k: io.StringIO(AMZN) if f == "/etc/os-release" else real(f, *a, **k)):
print(repr(dlc._retrieve_os()))
```
```
''
```
## Suggested fix
Parse `os-release` as the `key=value` format it is — split on the first `=`, strip surrounding quotes, take `ID` and `VERSION_ID` verbatim. That is a strict superset of the current regexes, so Ubuntu output is unchanged while Amazon Linux starts reporting `amzn2023`.
Worth noting separately: `_retrieve_os()` currently calls `open()` with no error handling, and `tag_instance()` calls it *before* its `try` block — so an unreadable `/etc/os-release` raises and kills the tagging process before it can tag at all.
PR: #6530
Contributor guide
Research direction
Start in scripts/docker/telemetry/deep_learning_container.py and run the supplied Python reproduction with the Amazon Linux /etc/os-release sample. Done means _retrieve_os() reports amzn2023 for Amazon Linux while preserving the existing Ubuntu result; PR #6530 indicates work is already underway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100