pybind / pybind/pybind11

[BUG]: std::ios breaks when using C++ module together with some Python modules

Open
#3,206 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Required prerequisites
Problem description

When importing a module made with pybind11 and some Python modules (scipy, pyarrow, matplotlib etc.), the "<<" operator breaks upon printing a float or an integer (stops printing anything). Printing strings does not cause the issue.

The bug occurs when the module is compiled together with a Xilinx shared library containing a C model of the soft decision FEC (libIp_sd_fec_v1_1_bitacc_cmodel.so). The following is what I observed:

  • When using the Xilinx shared library in a standalone C++ program, this does not occur.
  • When the C++ module linked with this library is imported before colliding Python modules (see above), std::ios functionality is maintained in that module only.
  • pybind11 modules imported after this module have the broken std::ios functionality (stop working after an integer or a float is printed).
  • stdio.h functionality is maintained
  • std::ios in std::cout and in std::ostringstream report both bad and fail bits after printing an integer or a float
  • When trying to use the std::ios in another process using the multiprocessing library, it works when the process start method is set to spawn instead of the default fork. I presume this is because the child process isn't linked with the libraries from the python modules when using spawn. Importing scipy or pyarrow in the child process causes the issue.

I have created a small demonstration code which reproduces the issue. I have tried to work around this by using the multiprocessing library to isolate the modules and stop this issue, but this slowed down the code significantly as overhead in copying large arrays from one process to another grew.

I use cppimport for compiling the code. Problem is reproduced when compiling the library manually also, so I don't think that is an issue.

I use Ubuntu 18.04 LTS with gcc version 7.5.0. Problem was reproduced both on python3.6 and python3.9.
The following is the minimal example code reproducing this issue:

test.py:

from scipy import *
import cppimport
import cppimport.import_hook
import test2

print(test2.testfcn2())

test2.cpp:

/*cppimport
<%
setup_pybind11(cfg)
cfg['libraries'] += ["Ip_sd_fec_v1_1_bitacc_cmodel"]
cfg['compiler_args'] += ['-g3', '-O0']
%>
*/

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>

#include <sstream>
#include <string.h>
#include <stdio.h>
#include <iostream>

// using namespace std;
namespace py = pybind11;
using namespace pybind11::literals;

std::string testfcn2(){
    std::ostringstream ss;
    
    std::printf("--- Starting the C++ stream testing ---");
    std::printf("\n Fail bits, ss: Good - %d Bad - %d EOF - %d Fail - %d\n", ss.good(), ss.bad(), ss.eof(), ss.fail());
    std::printf("\n Fail bits, cout: Good - %d Bad - %d EOF - %d Fail - %d\n", std::cout.good(), std::cout.bad(), std::cout.eof(), std::cout.fail());

    std::cout << "Printing from C++ code" << std::endl;
    std::cout << "Printing here also fails when printing a number: " 
        << 1  << std::endl;
    std::cout << "see?" << std::endl;
    
    std::printf("\n Fail bits, cout: Good - %d Bad - %d EOF - %d Fail - %d\n", std::cout.good(), std::cout.bad(), std::cout.eof(), std::cout.fail());
    

    ss << "abcd" <<  1 << "efg" << std::endl;
    ss << "abcd" <<  1 << "efg" << std::endl;
    
    std::printf("\n Fail bits, ss: Good - %d Bad - %d EOF - %d Fail - %d\n", ss.good(), ss.bad(), ss.eof(), ss.fail());
    
    return ss.str();
}

PYBIND11_MODULE(test2, m) {
    m.def("testfcn2", &testfcn2);
}

When commenting out the first line of the test.py file, the program prints this (which is normal):

--- Starting the C++ stream testing ---
 Fail bits, ss: Good - 1 Bad - 0 EOF - 0 Fail - 0

 Fail bits, cout: Good - 1 Bad - 0 EOF - 0 Fail - 0
Printing from C++ code
Printing here also fails when printing a number: 1
see?

 Fail bits, cout: Good - 1 Bad - 0 EOF - 0 Fail - 0

 Fail bits, ss: Good - 1 Bad - 0 EOF - 0 Fail - 0
abcd1efg
abcd1efg

When importing the scipy library, this happens:

--- Starting the C++ stream testing ---
 Fail bits, ss: Good - 1 Bad - 0 EOF - 0 Fail - 0

 Fail bits, cout: Good - 1 Bad - 0 EOF - 0 Fail - 0
Printing from C++ code
Printing here also fails when printing a number: 
 Fail bits, cout: Good - 0 Bad - 1 EOF - 0 Fail - 1

 Fail bits, ss: Good - 0 Bad - 1 EOF - 0 Fail - 1
abcd

I use the following versions of relevant packages:

  • cppimport==21.3.7
  • pybind11==2.7.1
  • scipy==1.7.1

I'm attaching the .so file. This is a proprietary file distributed by Xilinx.
libIp_sd_fec_v1_1_bitacc_cmodel.zip

Reproducible example code

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with test.py and test2.cpp, comparing behavior with and without the scipy import and the Xilinx shared library. Trace whether the stream-state corruption belongs to pybind11 or the external library; done requires a confirmed cause and a clearly scoped fix or documented external dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.