`SIGSEGV: invalid memory reference` in tests, but only when running them simultanously
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.2k
- Forks
- 1k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 66
Description
I'm facing an issue similar to #1257 but it only fails when I run multiple tests together, when running them one by one it's all good. I didn't open this PR as a bug report because I don't have the time right now to create a repro, once I'll have it I'll put it in the comments.
To make sure that it's not a happy little accident I ran the tests one by one a thousand times (5x1k runs, 0 failures) and I also ran all tests together (1k runs, 232 failures) so I can confidently say it's the tests running simultaneously that fails.
I also copied together my tests into a single one and run that as well which also produced no failures.
I also did a little experiment by just deleting/duplicating tests. When I only have 3 tests the error ratio is down to 6.2%, but when I have 15 tests there isn't a single one that ran successfully.
Another check I did was to add the serial_test library and annotate all tests with the #[serial] attribute which causes all tests but the first one fail with the following error: called `append_to_inittab` but a Python interpreter is already running which I don't really understand, my assumption would be that one executor stops before the other one starts, but I guess I'm wrong about this. I also tried running the tests with --jobs 1 but that also didn't help.
Basically all tests are the same, the only difference is that I'm loading different Python files:
#[test]
fn some_test() {
let source = include_str!("tests/some_test.py");
let data = load_some_data();
pyo3::append_to_inittab!(my_library);
Python::with_gil(|py| {
let globals = vec![("data", Py::new(py, data).unwrap())];
Python::run_bound(
py,
source,
Some(&globals.into_py_dict_bound(py)),
None,
)
})
.unwrap();
}
As a final clue, if I run multiple tests but comment out the pyo3::append_to_inittab!(my_library); line from all but a single test and cargo runs that test first then it's always green (I didn't do the 1k tests with it but run it manually ~20 times).
I cannot share the library I'm using as it's not open source, but based on these results I'm quite confident that the issue is not with the library itself. Sadly I don't have time right now to set up a minimal example with some 3rd party library.
Operating system and version: Ubuntu 22.04 (Linux Mint 21.3)
Python version: 3.10.12 (OS version)
Rust version: 1.78.0 (latest stable)
PyO3 version: originally faced the issue with 0.21, I updated to 0.22 but it's still present.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the concurrent Rust tests shown in the report, comparing repeated pyo3::append_to_inittab!(my_library) calls with a single call. Investigate PyO3's interpreter initialization and test execution behavior on Ubuntu 22.04 with Python 3.10, then establish a minimal reproducible example and a passing concurrent-test case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100