pytest-dev / pytest-dev/pytest

Bug with tolerances and other undesired behavior with approx using dict

Open
#8,703 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: approx
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Hi there,

Since I was introduced (yesterday) to the awesome feature of using approx with dictionaries I started to investigate its usage and I found some interesting things that I would like to share with you. I'm using pytest 6.2.2.

Let's start by showing what is perfectly working, so we have a baseline to the other cases:

 [1, 2, 3 + 1e-6] == pytest.approx([1, 2, 3], abs=1e-6)
>> False 

[1, 2, 3 + 1e-6]  == pytest.approx([1, 2, 3], abs=1e-5)
>> True

{'a': [1, 2, 3 + 1e-6]} == pytest.approx({'a': [1, 2, 3]}, abs=1e-6)
>> False

But here, the symmetry starts to break:

{'a': [1, 2, 3 + 1e-6]} == pytest.approx({'a': [1, 2, 3]}, abs=1e-5)
>> False

For brevity, I'm not showing here, but you can reproduce analogous issues with default tolerances.
I think this is a bug, but maybe you already have an opinion about this. I would like to know if I can help or if I should move on with another approach.

Beyond that, I also found some weird behavior with Numpy arrays. I know, this is even more tricky, but I would like to report some findings. Here we go:

{'a': np.array([1, 2, 3])} == pytest.approx({'a': [1, 2, 3]})
>> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

So, the numpy array (LHS) is being converted to a list and, well, maybe this could be considered a wrong usage, as the feature is very complex by itself. But there is also this case:

{'a': np.array([1, 2, 3])} == pytest.approx({'a': np.array([1, 2, 3])})
>> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Now I have two numpy arrays and both are converted to lists and are being compared directly with == operator, which gives us this error.

Any thoughts? If there's something I can help, I would love to. I just ask for some tips and I can prepare a PR.

Thank you!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the dictionary and NumPy examples in the issue, then trace the pytest.approx comparison path for dictionaries and array values. Done means the reported tolerance behavior is consistent and comparing NumPy arrays no longer raises an ambiguous truth-value error.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.