pytest-dev / pytest-dev/pytest

Inconsistent behavior between pytest [file_or_dir] and pytest / pytest -m marker / pytest -k expression

Open
#9,848 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

status: help wanted topic: collection
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Hello, I'm trying to write some pytests that would run some external python script using a mpirun based command through os.system(). For some reason, the command seems to be executed only when the tests are invoked with pytest [file_or_dir]. Invoking them with pytest, pytest -m marker, or pytest -k expression leads to the command exiting with exit code 1 and os.system returning 256.

Here's a simple minimal example to visualize the issue that I adapted from https://github.com/open-mpi/ompi/issues/6773 :
There are two python files, parallel.py, which contains the actions to be done by mpirun; and test_coder.py, which contains the function test_mpi() which calls parallel.py through mpi run.

parallel.py :

from mpi4py import MPI
comm = MPI.COMM_WORLD
print('RANK %s'%comm.rank)

test_coder.py :

import os, sys
import pytest

def test_mpi():
    print(sys.prefix)
    syspath = os.path.dirname(os.path.realpath(__file__))
    print('Running code')
    com = 'mpirun -v -np 3 ' + str(sys.executable) + " -m mpi4py " + syspath + "/parallel.py"
    err = os.system(com)
    print(err)
    print('Done')

The expected behavior, for example obtained when running python -c 'from test_coder import *; test_mpi()', is the following:

Running code
RANK 2
RANK 1
RANK 0
0
Done

When I run the test the following way: pytest -s test_coder.py, I get the correct output. However, if I run simply pytest, or pytest -s -k test_mpi, I get an incorrect output:

Running code
256
Done

Here, the command in os.system seems to have failed to run for some unknown reason. When running the test using a marker, pytest also behaves in this incorrect way. Is there anything in the difference in behaviour between pytest [file_or_dir] and pytest that could explain this inconsistency?

In https://github.com/open-mpi/ompi/issues/6773, the issue was resolved after recompiling mpi4py with the installed version of OMPI, however in my case this should not be the issue as it works perfectly fine outside using pytest and when using pytest [file_or_dir]. Could pytest somehow mess up with the envirionment (hence leading to the wrong mpi4py to be used)?

I am using Open MPI 4.0.3 on Ubuntu 20.04, with mpi4py==3.1.3 and pytest==7.1.1 (tried also with 6.2.5).

I would be very glad if you could give me any insight on this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the commands from test_coder.py with pytest, pytest -s test_coder.py, pytest -k test_mpi, and pytest -m marker, using parallel.py and the reported Open MPI, mpi4py, and pytest versions. Compare the environment and subprocess behavior for each invocation, then identify a pytest-side cause only if the difference is reproducible and define the expected consistent result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.