ProcessPoolExecutor deadlocks on on python 3.9 or newer

Starting in python 3.9.0, using a ProcessPoolExecutor has a good chance of deadlocking on a terra task. It's almost always happens with 10 workers, and is practically guaranteed with 16 workers.

I've managed to put together a piece of code to reproduce the error:

#!/usr/bin/env python

from concurrent.futures import ProcessPoolExecutor, as_completed

from celery import shared_task

@shared_task
def foo(x):
  return x*x

if __name__ == '__main__':
  futures = {}
  with ProcessPoolExecutor(max_workers=10) as executor:
    for x in range(10):
      futures[executor.submit(foo, x)] = x

    results = {}
    for future in as_completed(futures):
      task_id = futures[future]
      results[task_id] = future.result()
      print(len(results))

As you can see here, the bug is actually not part of terra, but can be reproduced just using the celery task object. Something about how celery works and a "task" vs a "function" is causing workers to hang before they ever process a single job.

Start with the supplied reproducer using ProcessPoolExecutor and the Celery shared_task object, then compare its behavior with a plain function on Python 3.9 or newer. Investigate why workers hang before processing jobs and verify the result with 10 and 16 workers; done means the reproducer completes without deadlocking.