adoptium / adoptium/installer

create_installer_linux is leaking files in /tmp

Open
#255 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jinja
Stars
168
Forks
77
Avg merge
2d 14h
Merged PRs (30d)
12

Description

Split out from https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1658

It looks like the Linux installer jobs are leaving files around in the `scripts` and `scripts-remapped` subdirectories of `/tmp/.gradle-test-kit-jenkins`. Each directory has between 10-20 files in, so are chewing that number of inodes per execution which made us hit the limit today. This needs to be resolved to stop the jobs chewing up all of the disk space over time.

```
jenkins@build-scaleway-ubuntu1604-x64-2:/tmp/.gradle-test-kit-jenkins/caches/6.1$ ls -latr scripts | tail -20
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 rdfmtqjtsjw79ydldswrdue0
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 5ehvfljxjsyas9hffm8rt0z8l
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 2vojtkti1yrozsqma34knrixw
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 dptk0e75u9a7wy6uxldkfjln8
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 14vo6ecxss0pyku7t71lig56n
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 5mstw6oal3khu68vhp73nmhyk
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 12:57 2ubvatqwwhqtdf4uzkq5w9pn0
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:07 ei3axmz0oamp6znebuh6tvy62
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:07 ciebxuayt1kj6inb0v8o022bx
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:07 9ep8wq1i3hscxcv0pcx4bkqw6
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:07 9dprf1hwuuef7av0c9q2r48eg
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:07 chikly6cxf5urp60m5d0ee3d8
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:07 27bgglemremg3xaejgwwbv9ie
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:43 5hm07pf01yuffxxalrxeg7mv5
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:44 2u89jabzxarrtty6qa1kvvcrr
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:44 r1qxqyma5227uz5um8yhfl5d
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:44 8khe6glg8y0vvfirn5zq7mynt
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:44 7egwqyemnwquhfk8hbzjmctw3
drwxrwxr-x 6114 jenkins jenkins 2953216 Nov 2 13:44 .
drwxrwxr-x 4 jenkins jenkins 4096 Nov 2 13:44 emwo3yz01ff29cehjfdglwld0
jenkins@build-scaleway-ubuntu1604-x64-2:/tmp/.gradle-test-kit-jenkins/caches/6.1$ ls -lart scripts-remapped | tail -20
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:07 settings_eyjao4bwa91uzxrvtg55r93o2
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:07 build_7kzhyfag3bckwei4b30re2ybd
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:07 settings_3xho9b0ae5zhpoz95tmqamyfw
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:07 build_5sjd1ijxqz1yjeyb8m6jv8t7x
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:07 settings_1azzikd60hpbumx4n2itb1tz1
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:07 build_erdksvbwihxdnf5cdhoyh2pjx
-rw-r--r-- 1 root root 3979712 Nov 2 13:28 cabbage
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:43 settings_davjwnc3gb8du4i2k6yxbsm08
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:43 build_d9phg9boq586lxvft2yxosm4s
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 settings_b4lpk77m1imwlisq9w82m16zz
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 build_acg6qjixed4klb54djgn3bs0s
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 settings_3z0m7n07ahfcagn3u83uugb44
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 build_6uw1wi4zm4e8ove39hftuzbbs
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 settings_48fo54553ssp7bzhzq7wlioh6
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 build_91i3es05l64m58qquzbrxl0pt
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 settings_2w32qfq3o309q1o93ztjuz3qq
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 build_4r8l4gjdl3nsr5s9ydq3589gk
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 settings_7wlnq78rfiyeywn6g55xi3dho
drwxrwxr-x 3 jenkins jenkins 4096 Nov 2 13:44 build_9glz09fu6r8pqr94iywtx6kir
drwxrwxr-x 1 jenkins jenkins 6868992 Nov 2 13:51 .
jenkins@build-scaleway-ubuntu1604-x64-2:/tmp/.gradle-test-kit-jenkins/caches/6.1$
```
I have mitigated it for now by deleting a few tens of thousands of the older directories (it started from about March 11th - possibly around the time it was last rebooted) but we should identify the cause and ensure it does not leave files around in the cache after each run.

Contributor guide

Open the contributing guide

Research direction

Start from the create_installer_linux job and inspect how its Gradle Test Kit work uses /tmp/.gradle-test-kit-jenkins, especially the scripts and scripts-remapped directories. Reproduce or trace one run and identify the cleanup boundary; done means repeated runs no longer leave cache files or directories accumulating there.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.