PyO3 / PyO3/pyo3

`SIGSEGV: invalid memory reference` in tests, but only when running them simultanously

Open
#4,298 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

I'm facing an issue similar to #1257 but it only fails when I run multiple tests together, when running them one by one it's all good. I didn't open this PR as a bug report because I don't have the time right now to create a repro, once I'll have it I'll put it in the comments.

To make sure that it's not a happy little accident I ran the tests one by one a thousand times (5x1k runs, 0 failures) and I also ran all tests together (1k runs, 232 failures) so I can confidently say it's the tests running simultaneously that fails.

I also copied together my tests into a single one and run that as well which also produced no failures.

I also did a little experiment by just deleting/duplicating tests. When I only have 3 tests the error ratio is down to 6.2%, but when I have 15 tests there isn't a single one that ran successfully.

Another check I did was to add the serial_test library and annotate all tests with the #[serial] attribute which causes all tests but the first one fail with the following error: called `append_to_inittab` but a Python interpreter is already running which I don't really understand, my assumption would be that one executor stops before the other one starts, but I guess I'm wrong about this. I also tried running the tests with --jobs 1 but that also didn't help.

Basically all tests are the same, the only difference is that I'm loading different Python files:

#[test]
fn some_test() {
    let source = include_str!("tests/some_test.py");

    let data = load_some_data();

    pyo3::append_to_inittab!(my_library);

    Python::with_gil(|py| {
        let globals = vec![("data", Py::new(py, data).unwrap())];

        Python::run_bound(
            py,
            source,
            Some(&globals.into_py_dict_bound(py)),
            None,
        )
    })
    .unwrap();
}

As a final clue, if I run multiple tests but comment out the pyo3::append_to_inittab!(my_library); line from all but a single test and cargo runs that test first then it's always green (I didn't do the 1k tests with it but run it manually ~20 times).

I cannot share the library I'm using as it's not open source, but based on these results I'm quite confident that the issue is not with the library itself. Sadly I don't have time right now to set up a minimal example with some 3rd party library.

Operating system and version: Ubuntu 22.04 (Linux Mint 21.3)
Python version: 3.10.12 (OS version)
Rust version: 1.78.0 (latest stable)
PyO3 version: originally faced the issue with 0.21, I updated to 0.22 but it's still present.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the concurrent Rust tests shown in the report, comparing repeated pyo3::append_to_inittab!(my_library) calls with a single call. Investigate PyO3's interpreter initialization and test execution behavior on Ubuntu 22.04 with Python 3.10, then establish a minimal reproducible example and a passing concurrent-test case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.