alunduil / alunduil/zfs-replicate
The mutation sweep's clean-test run passes Hypothesis's differing_executors health check
- Langage dominant
- Python
- Étoiles
- 24
- Forks
- 6
- Merge moyen
- 3 h 11 min
- PR mergées (30 j)
- 49
Description
## Summary
The nightly `Score mutant detection` job failed on 2026-08-24 before scoring anything: Hypothesis rejected `TestInits::test_length` with the `differing_executors` health check during the clean-test run. Find why the executor differs and fix it, rather than suppressing the check.
## Current behaviour
`daily.yml` runs `mutmut run --max-children 1`, which copies the tree to `mutants/` and runs the suite once unmutated as a baseline before scoring. On [run 32695453816](https://github.com/alunduil/zfs-replicate/actions/runs/32695453816) (2026-08-24, `master`, `a5f36c5`) that baseline failed:
```
...............F
=================================== FAILURES ===================================
____________________________ TestInits.test_length _____________________________
File "/home/runner/work/zfs-replicate/zfs-replicate/mutants/zfs_test/replicate_test/list_test.py", line 15, in test_length
def test_length(self, elements: List[int]) -> None:
hypothesis.errors.FailedHealthCheck: The method TestInits.test_length was called
from multiple different executors. This may lead to flaky tests and
nonreproducible errors when replaying from database.
...
1 failed, 15 passed in 0.16s
Failed to run clean test
```
The job stopped there and posted no score. `Daily` was green on every other day in the twelve to 2026-08-29, so this has fired once.
## Motivation
Hypothesis singles this health check out from the others:
> Unlike most health checks, `HealthCheck.differing_executors` warns about a correctness issue with your test. We therefore recommend fixing the underlying issue, rather than suppressing this health check.
So the cheap fix — adding `suppress_health_check=[HealthCheck.differing_executors]` — trades a red run for a test that may silently replay stale examples against the wrong instance. Worth understanding before choosing.
It also fails the sweep at the baseline, so a night that trips it produces no mutation score at all. #699's table goes stale without any signal that it did.
## Approach and alternatives
Not yet diagnosed. What is established:
- `zfs_test/replicate_test/list_test.py` puts `@given` on methods of `class TestInits`, the per-symbol class layout #486 introduced. Hypothesis's executor is the bound instance, so a class-based `@given` is the shape this check can fire on at all; module-level `@given` functions cannot trip it.
- The traceback names the `mutants/` copy, so at least two importable copies of the module exist during a sweep. Whether the two executors come from that duplication, from pytest re-instantiating the class, or from the `.hypothesis` example database being shared across both trees is unverified — that is the diagnosis this issue wants.
Reproduce it locally before picking a fix; a once-in-twelve-days failure suggests the trigger depends on run ordering or on database state carried between runs, either of which changes what the right fix is.
## Scope
- `zfs_test/replicate_test/list_test.py`
- `.github/workflows/daily.yml`, and whatever configures mutmut's clean-test run
- Hypothesis profile or example-database configuration, if the cause turns out to be there
## Out of scope
Making the sweep faster (#676) and the score itself (#699). This is only about the baseline run failing.
## Acceptance criteria
- [ ] The cause of the differing executors is identified and written down here
- [ ] `TestInits::test_length` runs under the mutation sweep without tripping the health check, by a fix rather than a suppression
- [ ] If suppression turns out to be the right call after all, the reason it is safe here is recorded
- [ ] The sweep reports a score, or fails loudly, when the clean-test run cannot complete
- [ ] Tests and pre-commit pass
## Related issues
#486 introduced the per-symbol test classes that put `@given` on methods. #676 covers the sweep's core usage and #699 carries the score table; neither touches the baseline failure.
Found while triaging default-branch CI failures across repositories for the fortnight to 2026-08-29.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Reproduce the clean-test run from .github/workflows/daily.yml, then inspect zfs_test/replicate_test/list_test.py and whatever configures mutmut and Hypothesis. Compare the executors and example-database behavior across the source tree and mutants/ copy. Done means the cause is documented, TestInits::test_length passes without suppressing the health check, the sweep reports a score or fails loudly, and tests and pre-commit pass.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- github-actions, python
- Domaine
- ci-cd, testing
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Active
- Clarté
- Plutôt claire
- Accessibilité débutants
- 48/100