kernelci / kernelci/kernelci-core

K8S: Non-existing docker images are consuming computing resources

Open
#1,618 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
120
Forks
107
Avg merge
1d 2h
Merged PRs (30d)
21

Description

I noticed it happens in past and now, that sometimes some docker images are overlooked and not created, as result if job with non-existing image is created, it will keep looping on k8s cluster endlessly, keeping nodes running (and consuming expensive computing resources).
For example:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 109s (x2242 over 8h) kubelet Back-off pulling image "kernelci/clang-11:riscv-kselftest-kernelci"
```

```
docker pull kernelci/clang-11:riscv-kselftest-kernelci
Error response from daemon: manifest for kernelci/clang-11:riscv-kselftest-kernelci not found: manifest unknown: manifest unknown
```

There are several ways to prevent such situations:
1)Before creating job just check if image exist by trivial docker pull or similar command. This have big drawback, we might hit limits.
2)During containers generation - create list of containers, put them on storage, and on each job generation verify, if such container exist, and if not - signal admins about error.

Contributor guide

No contributing guide indexed for this repository

Research direction

The report points to job creation and container generation as the relevant entry points; start by tracing how a requested image becomes a Kubernetes job. Compare the proposed image checks and container-list validation, then define which failure signal should prevent endless retries for the missing-image example.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.