matplotlib / matplotlib/pytest-mpl

Improving hash library workflow

Open
#158 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
272
Forks
53
PR merge metrics
No merged PRs in 30d

Description

For the current workflow, if there are hash library test failures for a pytest session, then these are easily discoverable thanks to the rich loveliness of the html figure comparison.

However, given any single hash library failure, the outcome is essentially three-fold:

  1. the graphic test has detected a genuine failure that requires to be resolved (somehow)
  2. the tolerance of the test requires to be increased
  3. a new hash value requires to be associated with the test and recorded in the hash library along with a new image

The choice of including or not including a new hash can result in a blend of the existing hash library and a new hash library (as opposed to accepting all new hashes).

We could extend the workflow to offer the user the choice to manually accept or reject any new proposed hashes. This would ease the burden of customising the hash library and also resolving any result-failed-diff.png images, all of which should then be placed under version control (somewhere).

We adopt this approach on SciTools/iris and SciTools/tephi, for example see https://github.com/SciTools/tephi/pull/78.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the pytest session workflow, the hash library handling, and the result-failed-diff.png artifacts described in the issue. Compare the approach used in SciTools/tephi pull request 78; done would mean users can manually accept or reject proposed hashes and the resulting hashes and images are kept under version control.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.