actions / actions/toolkit

Tool-cache: Support multiple runners sharing a single tool cache

Open
#804 7 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement tool-cache
Dominant language
TypeScript
Stars
5.9k
Forks
1.8k
No merged PRs in 30d

Description

Describe the bug
We have an issue when we try to use actions/tool-cache package in two different parallel running jobs. Jobs are allocated to a separate self hosted runner. It worth to mention our setup: self hosted runner is a pod running in the k8s cluster. Each pod runs only ones (it is being terminated right after the job allocated to this pod is done). There is a shared volume (shared between self hosted runners/pods) that is mount to each of the runners to the mount path of tool cache folder (RUNNER_TOOL_CACHE env). Meaning each pod can write to this folder (sometimes they do it simultaneously). In our case we use an actions that is called setup-java.
Workflow example:

name: pr-java
on: [pull_request]
jobs:
  build:
    runs-on: self-hosted
    steps:
        - uses: actions/checkout@v2
          with:
            ref: ${{ github.event.pull_request.head.sha }}
        - uses: actions/cache@v1
          with:
            path: ~/.m2/repository
            key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
            restore-keys: |
              ${{ runner.os }}-maven-
        - uses: actions/setup-java@v2
          with:
            java-version: 11
        - name: Docker build and push
          run: |
            mvn -B compile
            mvn -B -Dmaven.test.skip=true clean package
            mvn -B -Dmaven.test.skip=true -Dgit-revision=$(git rev-parse HEAD) dockerfile:build
            mvn -B -Dgit-revision=latest dockerfile:build
            mvn -B -Dgit-revision=$(git rev-parse HEAD) dockerfile:push
  test-unit:
    runs-on: self-hosted
    steps:
        - uses: actions/checkout@v2
          with:
            ref: ${{ github.event.pull_request.head.sha }}
        - uses: actions/cache@v1
          with:
            path: ~/.m2/repository
            key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
            restore-keys: |
              ${{ runner.os }}-maven-
        - uses: actions/setup-java@v2
          with:
            java-version: 11

Assuming that we run the workflow for the first time and there is no jdk stored inside tool cache folder. In this case we will have almost simultaneous write to cache folder because actions/setup-java@v2 step is used both in build and test-unit jobs.

Log snippets from both jobs follows:
build job, actions/setup-java step

##[debug]Evaluating condition for step: 'Run tradeshift/actions-setup-java@v1'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Run tradeshift/actions-setup-java@v1
##[debug]Register post job cleanup for action: tradeshift/actions-setup-java@v1
##[debug]Loading inputs
##[debug]Evaluating: secrets.MTLS_CACERT
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MTLS_CACERT'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_P12
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_P12'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_P12_PASSWORD
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_P12_PASSWORD'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_SETTINGS
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_SETTINGS'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_SECURITY
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_SECURITY'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run tradeshift/actions-setup-java@v1
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/build.properties
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/build.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/build.properties
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/build.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/build.properties
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/build.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/build.properties
##[debug]jdk extracted to /tmp/_temp/temp_1451549034/zulu11.48.21-ca-jdk11.0.11-linux_x64
##[debug]Caching tool jdk 11.0.11 x64
##[debug]source dir: /tmp/_temp/temp_1451549034/zulu11.48.21-ca-jdk11.0.11-linux_x64
##[debug]destination /var/cache/tools/jdk/11.0.11/x64
##[debug]finished caching tool
::set-output name=path::/var/cache/tools/jdk/11.0.11/x64
##[debug]='/var/cache/tools/jdk/11.0.11/x64'
::set-output name=version::11.0.11
##[debug]='11.0.11'

The test-unit actions/setup-java snippet:

##[debug]Evaluating condition for step: 'Run tradeshift/actions-setup-java@v1'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Run tradeshift/actions-setup-java@v1
##[debug]Register post job cleanup for action: tradeshift/actions-setup-java@v1
##[debug]Loading inputs
##[debug]Evaluating: secrets.MTLS_CACERT
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MTLS_CACERT'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_P12
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_P12'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_P12_PASSWORD
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_P12_PASSWORD'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_SETTINGS
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_SETTINGS'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating: secrets.MAVEN_SECURITY
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'MAVEN_SECURITY'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run tradeshift/actions-setup-java@v1
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/MemoryMonitor/build.properties
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/build.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/FullThreadDump/build.properties
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/build.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/VerboseGC/build.properties
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/build.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/jdk.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/project.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/file-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/nbproject/netbeans-targets.xml
zulu11.48.21-ca-jdk11.0.11-linux_x64/demo/nbproject/management/JTop/build.properties
##[debug]jdk extracted to /tmp/_temp/temp_1283980454/zulu11.48.21-ca-jdk11.0.11-linux_x64
##[debug]Caching tool jdk 11.0.11 x64
##[debug]source dir: /tmp/_temp/temp_1283980454/zulu11.48.21-ca-jdk11.0.11-linux_x64
##[debug]destination /var/cache/tools/jdk/11.0.11/x64
Error: Command failed: rm -rf "/var/cache/tools/jdk/11.0.11/x64"
rm: cannot remove '/var/cache/tools/jdk/11.0.11/x64/include': Directory not empty

##[debug]Node Action run completed with exit code 1

I suspect that failure happens here. It happens when two jobs simultaneously start to cache the tool. It looks like it is missing a locking mechanism. What is the desired behaviour in case tool cache folder is shared between runners? Should it be addressed in some other way ?

OS: Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the actions/tool-cache package operation shown by the concurrent setup-java logs, focusing on the cache destination under RUNNER_TOOL_CACHE. Reproduce two setup-java jobs writing the same JDK version concurrently; done means both jobs can populate or use the shared cache without removal or directory-not-empty failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.