alunduil / alunduil/zfs-replicate

The mutation sweep's clean-test run passes Hypothesis's differing_executors health check

Ouverte
#703 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug python
Langage dominant
Python
Étoiles
24
Forks
6
Merge moyen
3 h 11 min
PR mergées (30 j)
49

Description

## Summary

The nightly `Score mutant detection` job failed on 2026-08-24 before scoring anything: Hypothesis rejected `TestInits::test_length` with the `differing_executors` health check during the clean-test run. Find why the executor differs and fix it, rather than suppressing the check.

## Current behaviour

`daily.yml` runs `mutmut run --max-children 1`, which copies the tree to `mutants/` and runs the suite once unmutated as a baseline before scoring. On [run 32695453816](https://github.com/alunduil/zfs-replicate/actions/runs/32695453816) (2026-08-24, `master`, `a5f36c5`) that baseline failed:

```
...............F
=================================== FAILURES ===================================
____________________________ TestInits.test_length _____________________________
File "/home/runner/work/zfs-replicate/zfs-replicate/mutants/zfs_test/replicate_test/list_test.py", line 15, in test_length
def test_length(self, elements: List[int]) -> None:
hypothesis.errors.FailedHealthCheck: The method TestInits.test_length was called
from multiple different executors. This may lead to flaky tests and
nonreproducible errors when replaying from database.
...
1 failed, 15 passed in 0.16s
Failed to run clean test
```

The job stopped there and posted no score. `Daily` was green on every other day in the twelve to 2026-08-29, so this has fired once.

## Motivation

Hypothesis singles this health check out from the others:

> Unlike most health checks, `HealthCheck.differing_executors` warns about a correctness issue with your test. We therefore recommend fixing the underlying issue, rather than suppressing this health check.

So the cheap fix — adding `suppress_health_check=[HealthCheck.differing_executors]` — trades a red run for a test that may silently replay stale examples against the wrong instance. Worth understanding before choosing.

It also fails the sweep at the baseline, so a night that trips it produces no mutation score at all. #699's table goes stale without any signal that it did.

## Approach and alternatives

Not yet diagnosed. What is established:

- `zfs_test/replicate_test/list_test.py` puts `@given` on methods of `class TestInits`, the per-symbol class layout #486 introduced. Hypothesis's executor is the bound instance, so a class-based `@given` is the shape this check can fire on at all; module-level `@given` functions cannot trip it.
- The traceback names the `mutants/` copy, so at least two importable copies of the module exist during a sweep. Whether the two executors come from that duplication, from pytest re-instantiating the class, or from the `.hypothesis` example database being shared across both trees is unverified — that is the diagnosis this issue wants.

Reproduce it locally before picking a fix; a once-in-twelve-days failure suggests the trigger depends on run ordering or on database state carried between runs, either of which changes what the right fix is.

## Scope

- `zfs_test/replicate_test/list_test.py`
- `.github/workflows/daily.yml`, and whatever configures mutmut's clean-test run
- Hypothesis profile or example-database configuration, if the cause turns out to be there

## Out of scope

Making the sweep faster (#676) and the score itself (#699). This is only about the baseline run failing.

## Acceptance criteria

- [ ] The cause of the differing executors is identified and written down here
- [ ] `TestInits::test_length` runs under the mutation sweep without tripping the health check, by a fix rather than a suppression
- [ ] If suppression turns out to be the right call after all, the reason it is safe here is recorded
- [ ] The sweep reports a score, or fails loudly, when the clean-test run cannot complete
- [ ] Tests and pre-commit pass

## Related issues

#486 introduced the per-symbol test classes that put `@given` on methods. #676 covers the sweep's core usage and #699 carries the score table; neither touches the baseline failure.

Found while triaging default-branch CI failures across repositories for the fortnight to 2026-08-29.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Reproduce the clean-test run from .github/workflows/daily.yml, then inspect zfs_test/replicate_test/list_test.py and whatever configures mutmut and Hypothesis. Compare the executors and example-database behavior across the source tree and mutants/ copy. Done means the cause is documented, TestInits::test_length passes without suppressing the health check, the sweep reports a score or fails loudly, and tests and pre-commit pass.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
github-actions, python
Domaine
ci-cd, testing
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
48/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.