Persistent resource tracker after shutdown of ProcessPoolExecutor

Bug report

Bug description:

After launching jobs using a ProcessPoolExecutor instance and then shutting the instance down after they are complete, a subprocess launched by the executor to hold a semaphore lock is not shut down. This appears to be the reason why some jobs submitted to the batch queue of a distributed cluster are not terminated and have to be deleted manually.

I have confirmed the persistence of the process using the following code running on MacOS running Python 3.11 and Linux running Python 3.9.

import time

from concurrent.futures import ProcessPoolExecutor, as_completed

def test_function(i):
    time.sleep(20)
    return i

def test_pool():
    with ProcessPoolExecutor(max_workers=6) as executor:
        futures = []
        result = []
        for i in range(6):
            futures.append(executor.submit(test_function, i))
        for future in as_completed(futures):
            result.append(future.result())

    print(len(result))

if __name__ == '__main__':
    test_pool()

The complication is that, if you run this from the command line, the code will complete as expected. To see the problem, you have to embed the functions in an importable module and run the test_pool function in a debugger. I ran this in an interactive IPython shell within the NeXpy application.

Here are screenshots taken in VS Code, showing the processes before, during, and after running test_pool.

The issue is that I believe that the additional process (9812 in the above example) should be shut down when the executor's shutdown function is called. I have tried to modify the standard shutdown function to join and close the executor._call_queue and tried to release the executor._mp_context._rlock, which I think is what launches the additional process, but none of these shut it down.

CPython versions tested on:

3.9, 3.11

Operating systems tested on:

Linux, macOS

Reproduisez l’exemple fourni de ProcessPoolExecutor depuis un module importable sous un débogueur sur Linux ou macOS, puis inspectez shutdown(), _call_queue et _mp_context._rlock. C’est terminé lorsque le sous-processus avec verrou de sémaphore n’est plus présent après l’arrêt de l’exécuteur.