Persistent resource tracker after shutdown of ProcessPoolExecutor

Bug report

Bug description:

After launching jobs using a ProcessPoolExecutor instance and then shutting the instance down after they are complete, a subprocess launched by the executor to hold a semaphore lock is not shut down. This appears to be the reason why some jobs submitted to the batch queue of a distributed cluster are not terminated and have to be deleted manually.

I have confirmed the persistence of the process using the following code running on MacOS running Python 3.11 and Linux running Python 3.9.

import time

from concurrent.futures import ProcessPoolExecutor, as_completed

def test_function(i):
    time.sleep(20)
    return i

def test_pool():
    with ProcessPoolExecutor(max_workers=6) as executor:
        futures = []
        result = []
        for i in range(6):
            futures.append(executor.submit(test_function, i))
        for future in as_completed(futures):
            result.append(future.result())

    print(len(result))

if __name__ == '__main__':
    test_pool()

The complication is that, if you run this from the command line, the code will complete as expected. To see the problem, you have to embed the functions in an importable module and run the test_pool function in a debugger. I ran this in an interactive IPython shell within the NeXpy application.

Here are screenshots taken in VS Code, showing the processes before, during, and after running test_pool.

The issue is that I believe that the additional process (9812 in the above example) should be shut down when the executor's shutdown function is called. I have tried to modify the standard shutdown function to join and close the executor._call_queue and tried to release the executor._mp_context._rlock, which I think is what launches the additional process, but none of these shut it down.

CPython versions tested on:

3.9, 3.11

Operating systems tested on:

Linux, macOS

Tái hiện ví dụ ProcessPoolExecutor được cung cấp từ một module có thể import dưới debugger trên Linux hoặc macOS, sau đó kiểm tra shutdown(), _call_queue và _mp_context._rlock. Được xem là hoàn thành khi subprocess có khóa semaphore không còn tồn tại sau khi executor tắt.