python / python/cpython

multiprocessing.Process() with method fork() works not the same with os.fork() on unloading of .so files

Open
#127,213 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

I have a c/cpp file and compile it with pybind to get a so named xyz_test.cpython-311-x86_64-linux-gnu.so so that I can import it(xyz_test) and call a method named xyz_test.call() in Python code .

I use it with multiprocessing.Process() or os.fork(), but the unload process of this so is different:

with multiprocessing.setset_start_method('fork') and multiprocessing.Process(), this so is only unloaded while the parent process is terminated.
with os.fork(), this so is unloaded both while the child process and the parent process are terminated.

The behavior with os.fork() is the same with what I test on C code with C fork(), so is the behavior a bug with multiprocessing.Process() ?

Thank you!

# main_os_fork.py
import xyz_test
import os
import time

def func():
    pid = os.getpid()
    print(f'=== pid = {pid}, this is python func')
    xyz_test.callTLS()


if __name__ == '__main__':
    pid = os.getpid()
    print(f'=== pid = {pid}, this is python main')
    xyz_test.callTLS()
    fork_pid = os.fork()
    if fork_pid == 0:
        func()
    else:
        time.sleep(1)
# main_mp_fork.py
import xyz_test
import multiprocessing
import os
import time

def func():
    pid = os.getpid()
    print(f'=== pid = {pid}, this is python func')
    xyz_test.callTLS()
    time.sleep(10*60)


if __name__ == '__main__':
    pid = os.getpid()
    print(f'=== pid = {pid}, this is python main')
    xyz_test.callTLS()
    multiprocessing.set_start_method('fork')
    child = multiprocessing.Process(target=func)
    
    child.start()
    child.join()
# xyz_test.h
#ifndef SHARED_OBJ_H_
#define SHARED_OBJ_H_


#define PID_LOG(STR) \
auto pid = getpid(); \
printf("=== pid is %d, %s\n", pid, STR);

#define PRINT_ADDR() \
printf("\t now addr is %p\n", this);

void callTLS();

class A {
public:
    A();
    ~A();
    void call();
private:
    int a = 0;
};

#endif

# xyz_test.cc
#include "shared_obj.h"
#include <stdio.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <thread>
#include <iostream>

static thread_local A test;

void __attribute__((constructor)) init() {
    PID_LOG("so is loaded.");
}

void __attribute__((destructor)) fini() {
    PID_LOG("so is unloaded.");
}

A::A() {
    PID_LOG("this is A::A().");
    PRINT_ADDR();
}

A::~A() {
    PID_LOG("this is A::~A().");
    PRINT_ADDR();
}

void A::call() {
    PID_LOG("this is A::call().")
    PRINT_ADDR();
}

void callTLS() {
    test.call();
}
CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs
  • gh-127246

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with main_os_fork.py and main_mp_fork.py, then review the xyz_test.cc extension and compare their unload output on Linux with Python 3.11. Check linked PR gh-127246 and determine whether the differing .so teardown behavior is expected or requires a regression test and fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux, python
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.