POSIX multiprocessing spawn performance becomes 10x slower from a certain pickle size

Tái hiện sự khác biệt về thời gian bằng mp_pipe_limits.py trên Python 3.9 hoặc 3.10, sau đó kiểm tra multiprocessing/popen_spawn_posix.py xung quanh việc tạo pipe và reduction.dump. So sánh hành vi ở 65536 và 65537 byte, rồi xác định một bản sửa tránh được sự sụt giảm hiệu năng đột ngột; được coi là hoàn thành khi việc tái hiện không còn cho thấy hiện tượng chậm được báo cáo mà không cần dựa vào kích thước pipe cố định không an toàn.

Bug report

We are using multiprocessing with the spawn start method. On my 32-thread PC, starting all worker processes for my project used to take 2 seconds. At a certain point, it jumped straight to taking 20 seconds.

The slowdown appears as soon as more than 64 KB needs to be sent to a child process over the pipe.

Consider this minimal reproduction case:

#!/usr/bin/python3

import multiprocessing
import random
import sys
import time


class Container:

    def __init__(self, size):
        self.data = random.randbytes(size)


class ChildProcess(multiprocessing.Process):

    def __init__(self, name: str, container):
        super().__init__(name=name)
        self.container = container

    def run(self) -> None:
        print("Running")


def run():
    fixed_overhead_3_9 = 885
    difference = int(sys.argv[1])
    container = Container((64*1024) - fixed_overhead_3_9 + difference)
    children = [ChildProcess(f"child-{i}", container) for i in range(0, 2)]

    start_time = time.perf_counter()
    for child in children:
        child.start()
    end_time = time.perf_counter()

    print(f"Running took {int((end_time - start_time) * 1000)}ms")

    for child in children:
        child.join()


if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    run()

I added some "instrumentation" in multiprocessing/popen_spawn_posix.py to print the buffer size:

        try:
            reduction.dump(prep_data, fp)
            reduction.dump(process_obj, fp)
        finally:
            set_spawning_popen(None)
        print(len(fp.getbuffer()))
        parent_r = child_w = child_r = parent_w = None

Running the example results in:

$ python3.9 mp_pipe_limits.py 0
65536
65536
Running took 9ms
Running
Running
$ python3.9 mp_pipe_limits.py 1
65537
65537
Running
Running took 96ms
Running

Changing the pipe size with fcntl in multiprocessing/popen_spawn_posix.py restores performance:

        parent_r = child_w = child_r = parent_w = None
        try:
            parent_r, child_w = os.pipe()
            child_r, parent_w = os.pipe()
            fcntl.fcntl(parent_w, 1031, 100000)
            cmd = spawn.get_command_line(tracker_fd=tracker_fd,
                                         pipe_handle=child_r)

Where 1031 is fcntl.F_SETPIPE_SZ, which is not in Python 3.9.

Rerunning the reproduction case after this change:

$ python3.9 mp_pipe_limits.py 1
65537
65537
Running took 9ms
Running
Running

Of course, changing the pipe size will only delay the onset of the problem. The real solution (if there is any) will probably be different. Blindly setting a pipe size might also not be safe as it depends on limits set in /proc.

The example above is a best case example, since it has very limited pickle overhead. We hit this limit without any data caches involved. It's just our Python objects that live after application initialization. They are slower to pickle. However, then things are still 10x slower, so not just a fixed 80ms as seen in the example.

We use spawn instead of fork on Linux to avoid troubles with objects that cannot be pickled on other OSs (Windows).

Your environment

• CPython versions tested on: 3.9 and 3.10
• Operating system and architecture: Arch Linux (kernel 5.19.9-arch1-1), x86_64