kernelci / kernelci/kernelci-core
K8S: Non-existing docker images are consuming computing resources
- Dominant language
- Python
- Stars
- 120
- Forks
- 107
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 21
Description
I noticed it happens in past and now, that sometimes some docker images are overlooked and not created, as result if job with non-existing image is created, it will keep looping on k8s cluster endlessly, keeping nodes running (and consuming expensive computing resources).
For example:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 109s (x2242 over 8h) kubelet Back-off pulling image "kernelci/clang-11:riscv-kselftest-kernelci"
```
```
docker pull kernelci/clang-11:riscv-kselftest-kernelci
Error response from daemon: manifest for kernelci/clang-11:riscv-kselftest-kernelci not found: manifest unknown: manifest unknown
```
There are several ways to prevent such situations:
1)Before creating job just check if image exist by trivial docker pull or similar command. This have big drawback, we might hit limits.
2)During containers generation - create list of containers, put them on storage, and on each job generation verify, if such container exist, and if not - signal admins about error.
Contributor guide
No contributing guide indexed for this repository
Research direction
The report points to job creation and container generation as the relevant entry points; start by tracing how a requested image becomes a Kubernetes job. Compare the proposed image checks and container-list validation, then define which failure signal should prevent endless retries for the missing-image example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100