`RUNNER_TOOL_CACHE` and/or `AGENT_TOOLSDIRECTORY` not picked up on self hosted runner

Description:
I have exported RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY in the shell running the self-hosted runner, however a pip install in the workflow still uses /opt/hostedtoolcache directory.
job-log.txt

2024-07-23T09:20:33.0492190Z       [1/41] Compiling C object gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0493604Z       FAILED: gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0502062Z       cc -Igi/_gi.cpython-310-x86_64-linux-gnu.so.p -Igi -I../gi -I. -I.. -I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/
usr/include/gobject-introspection-1.0 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -DPY_SSIZE_T_CLEAN -Wall -Warray-bounds -Wcast-align -Wduplicated-branches -Wextra -Wform
at=2 -Wformat-nonliteral -Wformat-security -Wimplicit-function-declaration -Winit-self -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wmissing-prototypes -Wne
sted-externs -Wnull-dereference -Wold-style-definition -Wpacked -Wpointer-arith -Wrestrict -Wreturn-type -Wshadow -Wsign-compare -Wstrict-aliasing -Wstrict-prototypes -Wswitch-default -Wundef -Wunused-but-set-variable -Wwrite-strings -W
no-missing-field-initializers -Wno-unused-parameter -Wno-discarded-qualifiers -Wno-sign-conversion -Wno-cast-function-type -Wno-int-conversion -fno-strict-aliasing -fvisibility=hidden -MD -MQ gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -MF gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o.d -o gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -c ../gi/pygboxed.c
2024-07-23T09:20:33.0511885Z       cc1: warning: /opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10: No such file or directory [-Wmissing-include-dirs]
2024-07-23T09:20:33.0513997Z       ../gi/pygboxed.c:23:10: fatal error: Python.h: No such file or directory
2024-07-23T09:20:33.0515317Z          23 | #include <Python.h>
2024-07-23T09:20:33.0516190Z             |          ^~~~~~~~~~
2024-07-23T09:20:33.0517073Z       compilation terminated.

Action version:
actions/setup-python@v5

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:
python-version: '3.10'

Repro steps:

name: setup python on shr    
# Test pip install which requires compilation on host where include path is not correct    
# https://github.com/actions/setup-python/issues/485    
# https://github.com/actions/setup-python/issues/489    
    
on:    
  # Allows you to run this workflow manually from the Actions tab    
  workflow_dispatch:    
  push:    
    branches:    
      - setup-python-pip-install    
    
jobs:    
  test-python:    
    name: Test Python pip install    
    runs-on: [self-hosted, linux, test-python]    
    steps:    
      - name: Github Actions Context Information    
        uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7    
              
      - uses: actions/setup-python@v5    
        with:    
          python-version: '3.10'    
    
      - name: show env    
        run: |    
          env    
    
      - name: python pkg install    
        run: |    
          pip install PyGObject

From the Self-hosted runner directory:

export RUNNER_TOOL_CACHE="$(pwd)/_work/_tool"
export AGENT_TOOLSDIRECTORY="$(pwd)/_work/_tool"
./run.sh

Expected behavior:
I would expect the pip install step to set properly the -I flags without /opt/hostedtoolcache

Actual behavior:
The logs (see above) show that it tried to add an include path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which do not exist on the self hosted runner.

Workaround I am not fond of

export CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10
./run.sh

With this workaround I manage to install the lib, however it looks like that pip cached some stuff in the default cache directory:

  pip install PyGObject
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    PKG_CONFIG_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib/pkgconfig
    Python_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    Python2_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    Python3_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    LD_LIBRARY_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib
Collecting PyGObject
  Using cached pygobject-3.48.2.tar.gz (715 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pycairo>=1.16 (from PyGObject)
  Using cached pycairo-1.26.1-cp310-cp310-linux_x86_64.whl
Building wheels for collected packages: PyGObject
  Building wheel for PyGObject (pyproject.toml): started
  Building wheel for PyGObject (pyproject.toml): finished with status 'done'
  Created wheel for PyGObject: filename=pygobject-3.48.2-cp310-cp310-linux_x86_64.whl size=259354 sha256=fc1421f9adae4077d60198ee04309773969c33b2372d1cc93f2733967c3f3f96
  Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97
Successfully built PyGObject
Installing collected packages: pycairo, PyGObject
Successfully installed PyGObject-3.48.2 pycairo-1.26.1

(Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97)
even though the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY have been set up.

I took the idea of this workaround from https://github.com/actions/setup-python/issues/489 and https://github.com/actions/setup-python/issues/485
However I do not understand why the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY are not used.

Beside, if i run

grep -R /opt/hostedtoolcache *

from _work/_tool I get a lot of match:

...
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIGURE_LDFLAGS=      -Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:prefix=         /opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIG_ARGS=     '--prefix=/opt/hostedtoolcache/Python/3.10.14/x64' '--enable-shared' '--enable-optimizations' '--enable-loadable-sqlite-extensions' 'LDFLAGS=-Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib'
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/python-config.py:#!/opt/hostedtoolcache/Python/3.10.14/x64/bin/python3.10
grep: Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.a: binary file matches
Python/3.10.14/x64/lib/pkgconfig/python-3.10-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python-3.10.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
...

I don't really know where pip gets the include path from, but I guess from one of the above files which clearly points to an invalid folder.

Thanks in advance for your help, let me know if you need more information.

Reproduce the provided self-hosted Ubuntu workflow with actions/setup-python@v5 and inspect the installed Python metadata under _work/_tool, including its Makefile, pkg-config files, and python-config.py. Trace where pip obtains the /opt/hostedtoolcache include path; done means the workflow uses the self-hosted tool path and pip can build PyGObject without CPATH.