Persistent resource tracker after shutdown of ProcessPoolExecutor

Bug report

Bug description:

After launching jobs using a ProcessPoolExecutor instance and then shutting the instance down after they are complete, a subprocess launched by the executor to hold a semaphore lock is not shut down. This appears to be the reason why some jobs submitted to the batch queue of a distributed cluster are not terminated and have to be deleted manually.

I have confirmed the persistence of the process using the following code running on MacOS running Python 3.11 and Linux running Python 3.9.

import time

from concurrent.futures import ProcessPoolExecutor, as_completed

def test_function(i):
    time.sleep(20)
    return i

def test_pool():
    with ProcessPoolExecutor(max_workers=6) as executor:
        futures = []
        result = []
        for i in range(6):
            futures.append(executor.submit(test_function, i))
        for future in as_completed(futures):
            result.append(future.result())

    print(len(result))

if __name__ == '__main__':
    test_pool()

The complication is that, if you run this from the command line, the code will complete as expected. To see the problem, you have to embed the functions in an importable module and run the test_pool function in a debugger. I ran this in an interactive IPython shell within the NeXpy application.

Here are screenshots taken in VS Code, showing the processes before, during, and after running test_pool.

The issue is that I believe that the additional process (9812 in the above example) should be shut down when the executor's shutdown function is called. I have tried to modify the standard shutdown function to join and close the executor._call_queue and tried to release the executor._mp_context._rlock, which I think is what launches the additional process, but none of these shut it down.

CPython versions tested on:

3.9, 3.11

Operating systems tested on:

Linux, macOS

Reproduzca el ejemplo proporcionado de ProcessPoolExecutor desde un módulo importable bajo un depurador en Linux o macOS y, a continuación, inspeccione shutdown(), _call_queue y _mp_context._rlock. Se considera completado cuando el subproceso con bloqueo de semáforo ya no está presente después del apagado del executor.