Self-hosted runners cache/reuse OS-incompatible python installations

Description:
Our GitHub Actions runners are self-hosted.

Some of our CI jobs run in different operating systems using the Run jobs in a container functionality, for example if they need a specific OS, or are testing on different operating systems in a test matrix.

When running the actions/setup-python action, it will either download Python and cache it, or use the existing cached python installation.

The problem is, while the download URL is an OS-specific download, the cache directory is not OS-specific. For example, these python downloads are different, but both get cached in /opt/actions-runner/_work/_tool/Python/3.8.18/x64.

• https://github.com/actions/python-versions/releases/download/3.8.18-12303122501/python-3.8.18-linux-24.04-x64.tar.gz
• https://github.com/actions/python-versions/releases/download/3.8.18-12303122501/python-3.8.18-linux-20.04-x64.tar.gz

The cache destination gets reused across jobs irrespective of what operating system the job is running on, which leads to the python installation breaking.

Example

• Job 1. Running on Ubuntu 20.04 container

Cache miss, python gets downloaded and cached...

  Version 3.8 was not found in the local cache
  Version 3.8 is available for downloading
  Download from "https://github.com/actions/python-versions/releases/download/3.8.18-12303122501/python-3.8.18-linux-20.04-x64.tar.gz"
  Extract downloaded archive
  /usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /__w/_temp/bba86c99-5e24-4327-a571-cb48396a8d3a -f /__w/_temp/568681cf-5876-4ee5-b706-acfdd405f2f3
  Execute installation script
  Check if Python hostedtoolcache folder exist...
  Creating Python hostedtoolcache folder...
  Create Python 3.8.18 folder
  Copy Python binaries to hostedtoolcache folder
  Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
  Upgrading pip...
  Looking in links: /tmp/tmpmip5jefu
  Requirement already satisfied: setuptools in /__w/_tool/Python/3.8.18/x64/lib/python3.8/site-packages (56.0.0)
  Requirement already satisfied: pip in /__w/_tool/Python/3.8.18/x64/lib/python3.8/site-packages (23.0.1)
  Collecting pip
  Downloading pip-25.0.1-py3-none-any.whl (1.8 MB)
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 9.8 MB/s eta 0:00:00
  
  Installing collected packages: pip
  Attempting uninstall: pip
  Found existing installation: pip 23.0.1
  Uninstalling pip-23.0.1:
  Successfully uninstalled pip-23.0.1
  Successfully installed pip-25.0.1
  Create complete file
  Successfully set up CPython (3.8.18)

• Job 2. Running in Ubuntu 24.04 container on the same runner

Cache hit (of the incorrect OS's python!):

Run actions/setup-python@v5
  with:
    python-version: 3.8
    check-latest: false
    token: ***
    update-environment: true
    allow-prereleases: false
    freethreaded: false
Installed versions
  Successfully set up CPython (3.8.18)

Run python -m pip install --upgrade pip
  python -m pip install --upgrade pip
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/actions-runner/_work/_tool/Python/3.8.18/x64
    PKG_CONFIG_PATH: /opt/actions-runner/_work/_tool/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/actions-runner/_work/_tool/Python/3.8.18/x64
    Python2_ROOT_DIR: /opt/actions-runner/_work/_tool/Python/3.8.18/x64
    Python3_ROOT_DIR: /opt/actions-runner/_work/_tool/Python/3.8.18/x64
    LD_LIBRARY_PATH: /opt/actions-runner/_work/_tool/Python/3.8.18/x64/lib
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Requirement already satisfied: pip in /opt/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/site-packages (25.0.1)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

The above error is likely due Ubuntu 20 and Ubuntu 24 using incompatible/different OpenSSL versions (1 vs 3).

The same can happen in the other direction too, if the Ubuntu 24.04 job runs before the Ubuntu 20.04 job, then I encountered this issue:

Run python -m pip install --upgrade pip
python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by python)
python: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /__w/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0)
python: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /__w/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0)
python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /__w/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0)
python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /__w/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0)
python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /__w/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0)
python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /__w/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0)

In this case, because the python release for Ubuntu 24.04 is dynamically linked against a glibc version that isn't installed on Ubuntu 20.04.

Action version:
actions/setup-python@v5

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:
I expect all python versions may be affected

Repro steps:

1. Create a single self-hosted Github Actions runner
2. Run a CI job that is using Ubuntu 24.04
   i. Run actions/setup-python@v5 targeting python 3.8
3. Wait for above job to finish
4. Run a CI job that is using Ubuntu 20.04.
   i. Run actions/setup-python@v5 targeting python 3.8
   ii. Run python -m pip install --upgrade pip

The above should fail. The python installation is broken as it is incompatible with that Ubuntu version.

The problem can be reproduced if you swap the operating systems around too (Ubuntu 20 first, then Ubuntu 24), but a different python error will happen.

Expected behavior:

If an OS-specific python release is utilised in the download, then this should be cached into an OS-specific location on the CI runner to ensure an equivalent cached python release is utilised.

Actual behavior:
In a multi-OS environment using actions/setup-python it is likely GitHub Actions will cache an incompatible python installation.

Bắt đầu với phần triển khai của actions/setup-python@v5 và tái hiện các container job Ubuntu 24.04 và Ubuntu 20.04 được liệt kê trên một self-hosted runner duy nhất. Theo dõi cách vị trí Python tool-cache hiện có được chọn; công việc hoàn tất khi mỗi bản cài đặt dành riêng cho từng hệ điều hành được cache riêng và việc tái hiện không còn sử dụng lại một bản build Python không tương thích.