pytest-dev / pytest-dev/pytest
Inconsistent behavior between pytest [file_or_dir] and pytest / pytest -m marker / pytest -k expression
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
Hello, I'm trying to write some pytests that would run some external python script using a mpirun based command through os.system(). For some reason, the command seems to be executed only when the tests are invoked with pytest [file_or_dir]. Invoking them with pytest, pytest -m marker, or pytest -k expression leads to the command exiting with exit code 1 and os.system returning 256.
Here's a simple minimal example to visualize the issue that I adapted from https://github.com/open-mpi/ompi/issues/6773 :
There are two python files, parallel.py, which contains the actions to be done by mpirun; and test_coder.py, which contains the function test_mpi() which calls parallel.py through mpi run.
parallel.py :
from mpi4py import MPI
comm = MPI.COMM_WORLD
print('RANK %s'%comm.rank)
test_coder.py :
import os, sys
import pytest
def test_mpi():
print(sys.prefix)
syspath = os.path.dirname(os.path.realpath(__file__))
print('Running code')
com = 'mpirun -v -np 3 ' + str(sys.executable) + " -m mpi4py " + syspath + "/parallel.py"
err = os.system(com)
print(err)
print('Done')
The expected behavior, for example obtained when running python -c 'from test_coder import *; test_mpi()', is the following:
Running code
RANK 2
RANK 1
RANK 0
0
Done
When I run the test the following way: pytest -s test_coder.py, I get the correct output. However, if I run simply pytest, or pytest -s -k test_mpi, I get an incorrect output:
Running code
256
Done
Here, the command in os.system seems to have failed to run for some unknown reason. When running the test using a marker, pytest also behaves in this incorrect way. Is there anything in the difference in behaviour between pytest [file_or_dir] and pytest that could explain this inconsistency?
In https://github.com/open-mpi/ompi/issues/6773, the issue was resolved after recompiling mpi4py with the installed version of OMPI, however in my case this should not be the issue as it works perfectly fine outside using pytest and when using pytest [file_or_dir]. Could pytest somehow mess up with the envirionment (hence leading to the wrong mpi4py to be used)?
I am using Open MPI 4.0.3 on Ubuntu 20.04, with mpi4py==3.1.3 and pytest==7.1.1 (tried also with 6.2.5).
I would be very glad if you could give me any insight on this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the commands from test_coder.py with pytest, pytest -s test_coder.py, pytest -k test_mpi, and pytest -m marker, using parallel.py and the reported Open MPI, mpi4py, and pytest versions. Compare the environment and subprocess behavior for each invocation, then identify a pytest-side cause only if the difference is reproducible and define the expected consistent result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100