mesonbuild / mesonbuild/meson-python

pytest assertion rewriting does not seem to work with meson editable install

Open
#646 4 comments 1 reaction 0 assignees View on GitHub
dependency-bug
Dominant language
Python
Stars
180
Forks
93
Avg merge
2d 7h
Merged PRs (30d)
18

Description

It could well be partially linked to some of the discussion in https://github.com/mesonbuild/meson-python/issues/642 but I thought a separate issue may be worth it.

### Description of the issue

The pytest assertion rewriting that shows additional debug information when an assert fails does not show up when using meson editable install. It does work fine when using normal (non-editable) meson install. Although not crucial I would argue that this can be actually quite useful in CI logs.

It could also well be a Pytest issue but I thought I had to start somewhere and chose meson-python.

More background about Pytest assertion rewriting, see [Pytest doc](https://docs.pytest.org/en/stable/how-to/assert.html#assertion-introspection-details) that links to a somewhat old [blog post](http://pybites.blogspot.com/2011/07/behind-scenes-of-pytests-new-assertion.html) but I am guessing is probably still relevant?

This was originally seen in scikit-learn (see https://github.com/scikit-learn/scikit-learn/issues/29253#issuecomment-2173549242 if you are really curious where it would have actually been quite useful to see intermediary values in the `AssertionError` in https://github.com/scikit-learn/scikit-learn/issues/29253#issuecomment-2167610891). I put together a reproducer below with numpy hoping this makes it a bit easier to look at. I guess a smaller reproducer could be put together if deemed useful, but it would take a bit of time (I tried with https://github.com/dnicolodi/python-siphash24 and I am not able to reproduce the issue).

### Get the reproducer from my numpy branch

```
git clone https://github.com/lesteve/numpy
cd numpy
git checkout meson-editable-pytest-assertion-rewrite
```

I added a simple test in `numpy/tests/test_mine.py`
```
import numpy as np

def test():
arr = np.arange(100)
assert arr.sum() < 100
```

When running with pytest you pytest assertion rewriting shows the value or `arr.sum()` (see below).

### Normal (non-editable) install

```
pip install --verbose --no-build-isolation .
```

Note the assertion rewriting i.e. you get the value of `arr.sum()`

Partial output:
```
E assert np.int64(4950) < 100
E + where np.int64(4950) = ()
```

Full output
```
❯ pytest --pyargs numpy.tests.test_mine
========================== test session starts ===========================
platform linux -- Python 3.11.7, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/lesteve
plugins: hypothesis-6.103.2
collected 1 item

micromamba/envs/numpy-dev/lib/python3.11/site-packages/numpy/tests/test_mine.py F [100%]

================================ FAILURES ================================
__________________________________ test __________________________________

def test():
arr = np.arange(100)
> assert arr.sum() < 100
E assert np.int64(4950) < 100
E + where np.int64(4950) = ()
E + where = array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,\n 17, 18, 19, 20, 21, 22, 23, 24, 25, ...72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,\n 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]).sum

micromamba/envs/numpy-dev/lib/python3.11/site-packages/numpy/tests/test_mine.py:6: AssertionError
======================== short test summary info =========================
FAILED micromamba/envs/numpy-dev/lib/python3.11/site-packages/numpy/tests/test_mine.py::test - assert np.int64(4950) < 100
=========================== 1 failed in 0.82s ============================

```

### Editable install

```
pip install --verbose --no-build-isolation --editable . --config-settings editable-verbose=true
```

You don't get the value of `arr.sum()`
```
pytest --pyargs numpy.tests.test_mine
========================== test session starts ===========================
platform linux -- Python 3.11.7, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/lesteve
plugins: hypothesis-6.103.2
collecting ... + /home/lesteve/micromamba/envs/numpy-dev/bin/ninja
[1/1] Generating numpy/generate-version with a custom command
Saving version to numpy/version.py
collected 1 item

dev/numpy/numpy/tests/test_mine.py F [100%]

================================ FAILURES ================================
__________________________________ test __________________________________

def test():
arr = np.arange(100)
> assert arr.sum() < 100
E AssertionError

dev/numpy/numpy/tests/test_mine.py:6: AssertionError
======================== short test summary info =========================
FAILED dev/numpy/numpy/tests/test_mine.py::test - AssertionError
=========================== 1 failed in 1.02s ============================
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproducer in numpy/tests/test_mine.py and compare pytest output after the documented normal and editable install commands. Trace how the editable installation affects pytest assertion rewriting, using the linked Meson issue for context. Done means the editable-install run reports the evaluated assertion values, matching the normal-install behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.