kernelci / kernelci/kernelci-pipeline

Docker images cleanup

Open
#421 0 comments 0 reactions 0 assignees View on GitHub
enhancement techdebt
Dominant language
Python
Stars
11
Forks
40
Avg merge
2d 13h
Merged PRs (30d)
14

Description

As part of preparation to versioned API/Pipeline release we need to create mechanism that will delete images that are old and not used anymore.
As first step we might do semi-automatic cleanup of existing images.
For example i took snapshots of images with following format at 7 and 8 February: "imagename,last_update,num_pulls" and identified several stale images that was not updated and dont have any new pulls:
```
2 build-base,2022-08-15T14:46:00.107953Z,3568751
2 build-clang-10,2021-10-11T09:30:23.692938Z,24702
2 build-clang-11,2021-10-11T09:31:52.344053Z,3657
2 build-clang-12,2021-10-11T09:33:21.871013Z,4381
2 build-clang-8,2020-02-07T13:33:10.186608Z,16015
2 build-clang-9,2020-09-11T12:24:23.794136Z,4115
2 build-gcc-6.3.0_mips,2019-01-02T19:22:41.424879Z,74641
2 build-gcc-7_arc,2019-04-26T12:07:11.857286Z,44494
2 build-gcc-7_arm,2019-04-26T12:07:15.222898Z,422144
2 build-gcc-7_arm64,2019-04-26T12:07:18.605918Z,22994
2 build-gcc-7_mips,2019-04-26T12:07:21.970747Z,150251
2 build-gcc-7_riscv64,2019-04-26T12:07:25.41024Z,12862
2 build-gcc-7_x86,2019-04-26T12:07:28.428278Z,26355
2 build-gcc-8_arc,2021-10-11T09:27:48.156805Z,416342
2 build-gcc-8_arm,2021-10-11T09:28:03.844757Z,2739324
2 build-gcc-8_arm64,2021-10-11T09:28:24.603019Z,200543
2 build-gcc-8_mips,2021-10-11T09:28:37.372275Z,1236382
2 build-gcc-8_riscv64,2021-10-11T09:28:49.307957Z,146526
2 build-gcc-8_x86,2021-10-11T09:28:58.174458Z,281651
2 build-k8s,2021-10-11T09:36:37.426002Z,2266357
2 buildroot,2023-10-06T19:01:55.263699Z,1231
2 chromeos-flash,2022-05-23T02:03:33.745155Z,158
2 chromeos-lavapost,2022-05-20T02:06:25.633994Z,338
2 chromeos-tast,2022-05-20T02:06:35.408823Z,888
2 chromiumos,2022-05-20T02:06:42.177093Z,222
2 clang-10,2021-12-13T22:50:40.672498Z,9773
2 clang-11,2024-02-06T11:02:42.852681Z,11597
2 clang-12,2024-02-06T10:21:35.064746Z,111
2 clang-13,2024-02-06T10:24:38.197558Z,10963
2 clang-14,2024-02-06T11:26:30.607968Z,84417
2 clang-15,2024-02-06T11:50:22.579164Z,7006
2 clang-17,2024-02-06T12:40:57.498448Z,4122
2 clang-base,2022-08-15T14:58:02.019422Z,62
2 coccinelle,2022-08-15T14:58:12.506947Z,15
2 cros-clang-11,2023-02-01T15:25:45.894674Z,4
2 cros-clang-14,2023-08-25T07:20:37.6602Z,7785
2 cros-clang-16,2023-02-01T15:26:15.328995Z,4
2 cvehound,2024-02-06T14:28:44.7563Z,29
2 dt-validation,2024-02-06T14:29:59.817239Z,312
2 gcc-10,2024-02-06T13:14:37.265856Z,274361
2 gcc-10_arc,2022-08-15T14:49:03.747169Z,183591
2 gcc-10_arm,2022-08-15T14:49:38.493312Z,499481
2 gcc-10_arm64,2022-08-15T14:50:44.181487Z,63658
2 gcc-10_armv5,2022-08-15T14:51:19.350513Z,1
2 gcc-10_mips,2022-08-15T14:51:34.29928Z,246272
2 gcc-10_riscv64,2022-08-15T14:51:47.765388Z,52263
2 gcc-10_sparc,2022-08-15T14:52:02.525232Z,435
2 gcc-10_x86,2022-08-15T14:52:16.083198Z,72443
2 jenkins-agent,2020-04-06T20:33:56.674786Z,17
2 k8s,2024-02-06T10:15:07.566997Z,3379028
2 kcidb,2020-08-27T12:34:58.106175Z,22
2 kernelci,2024-02-06T10:09:35.067404Z,33583
2 pipeline,2023-09-05T08:17:46.959999Z,56
2 qemu,2024-02-06T14:27:20.079026Z,331743
2 rust,2023-10-06T17:58:23.886998Z,0
2 rustc-1.62,2023-07-17T18:46:47.731383Z,2702
2 rustc-1.66,2023-07-17T19:41:01.815591Z,37
2 rustc-1.68,2023-04-24T09:39:11.238997Z,0
2 rustc-1.70,2023-08-22T20:33:51.302541Z,285
2 rustc-1.71,2023-10-02T11:07:01.41494Z,374
2 rustc-1.72,2023-10-09T16:20:47.071371Z,123
2 rustc-1.73,2023-12-21T22:58:16.359128Z,323
2 rustc-1.74,2024-02-06T14:26:17.171256Z,14
2 staging-build-base,2022-08-04T00:07:23.254115Z,110375
2 staging-build-clang-8,2020-01-22T12:30:00.472868Z,1005
2 staging-build-clang-9,2020-09-15T08:04:42.435954Z,4512
2 staging-build-gcc-6.3.0_mips,2018-11-07T17:36:41.482595Z,30
2 staging-build-gcc-7_arm,2018-11-07T17:12:42.25613Z,24
2 staging-build-gcc-7_arm64,2018-11-07T17:17:28.311113Z,29
2 staging-build-gcc-7_mips,2018-11-07T17:26:30.366179Z,31
2 staging-build-gcc-7_x86,2018-11-07T17:21:50.792404Z,31
2 staging-build-gcc-8_arc,2021-09-17T06:06:21.600348Z,52204
2 staging-build-gcc-8_arm,2021-09-17T06:06:24.010075Z,383214
2 staging-build-gcc-8_arm64,2021-09-17T06:06:26.570485Z,44352
2 staging-build-gcc-8_mips,2021-09-17T06:06:28.90399Z,84833
2 staging-build-gcc-8_riscv64,2021-09-17T06:06:31.258531Z,23326
2 staging-build-gcc-8_x86,2021-09-17T06:06:33.69435Z,44969
2 staging-build-gcc-9_mips,2021-09-16T20:30:58.705415Z,0
2 staging-build-k8s,2021-09-29T08:23:56.018989Z,55560
2 staging-chromiumos,2022-05-20T02:06:49.420492Z,26
2 staging-clang-10,2021-12-02T08:05:25.048825Z,18339
2 staging-clang-11,2023-04-04T16:11:38.071066Z,8490
2 staging-clang-12,2022-08-04T00:10:47.071355Z,567
2 staging-clang-13,2022-08-04T00:11:43.920914Z,5219
2 staging-clang-15,2022-08-18T08:31:08.322704Z,2123
2 staging-clang-16,2023-02-13T08:17:28.268825Z,2423
2 staging-clang-base,2022-08-04T00:08:33.596014Z,436
2 staging-coccinelle,2022-08-18T08:54:25.927787Z,669
2 staging-cros-baseline,2023-05-30T13:23:54.412257Z,1
2 staging-cros-clang-14,2023-05-30T13:16:42.832196Z,4
2 staging-cros-gcc-10,2023-05-30T13:20:18.495393Z,3
2 staging-cros-k8s,2023-05-30T13:12:55.110666Z,1
2 staging-cros-kernelci,2023-05-30T13:10:28.704068Z,3
2 staging-cros-qemu-modules,2023-05-30T13:26:20.895741Z,1
2 staging-cros-sdk,2023-05-30T13:22:42.146207Z,1
2 staging-cros-tast,2023-05-30T13:27:42.403389Z,1
2 staging-deploy,2019-11-08T10:21:31.935004Z,21
2 staging-gcc-10_arc,2022-08-04T00:08:36.152499Z,3701
2 staging-gcc-10_arm,2022-08-04T00:08:39.048897Z,14893
2 staging-gcc-10_arm64,2022-08-04T00:08:41.999232Z,8803
2 staging-gcc-10_armv5,2022-08-04T00:08:44.912545Z,1
2 staging-gcc-10_mips,2022-08-04T00:08:47.37552Z,7370
2 staging-gcc-10_riscv64,2022-08-04T00:08:50.164073Z,3563
2 staging-gcc-10_sparc,2022-08-04T00:08:58.375326Z,6
2 staging-gcc-10_sparc64,2022-08-03T00:06:47.978503Z,58
2 staging-gcc-10_x86,2022-08-04T00:09:01.313404Z,23985
2 staging-rustc-1.62,2023-04-21T11:14:47.483571Z,210
2 staging-rustc-1.66,2023-04-04T16:30:36.98395Z,35
2 staging-rustc-1.68,2023-07-20T16:04:52.460948Z,105
2 staging-rustc-1.70,2023-07-29T16:39:13.415872Z,10
2 staging-rustc-1.71,2023-10-06T11:34:47.375281Z,47
2 staging-rustc-1.72,2023-10-09T09:06:23.303455Z,10
2 staging-rustc-1.73,2023-12-14T16:34:26.261662Z,93
```
Likely i will need to take larger time span, such as week.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the repository entry points responsible for publishing or managing Docker images and determine how image age and pull counts can be collected over a longer period. Define the retention criteria and whether cleanup is semi-automatic or fully automatic. Done means stale, unused images can be identified and removed without affecting images still needed by the versioned API/Pipeline release.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.