test_snap_versions lint failure due to snapshot key ordering mismatch (and stale snapshot persistence)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 322
- Forks
- 255
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
Description of the bug
Description
I have encountered two related behaviors regarding nf-test snapshots and nf-core modules lint that lead to false failures and stale data management issues.
Bug 1: Snapshot Ordering Mismatch causes Lint Failure
When a module has multiple tests (e.g., a script test and a stub test), the order in which entries appear in the *.nf.test.snap file can drift from the order in main.nf.test. This causes nf-core modules lint to report missing versions, even when they exist.
Steps to Reproduce:
Create a module with two tests in main.nf.test:
Test A: A "real" script test (initially configured without snapshotting output/versions).
Test B: A stub test (configured with assertions/snapshots).
Run nf-test test.
Result: main.nf.test.snap is created containing only Test B data.
Modify Test A to assert/snapshot versions.yml.
Run nf-test test again.
Result: Test A data is appended to the bottom of the existing main.nf.test.snap.
Current File State:
main.nf.test: Order is [Test A, Test B].
main.nf.test.snap: Order is [Test B, Test A].
Run nf-core modules lint.
Observed Behavior:
The linter fails with the error:
test_snap_versions: versions not found in snapshot
This occurs because the linter appears to expect the snapshot keys to appear in a specific order or struggles to parse the versions when the script test appears after the stub test in the snapshot file.
Expected Behavior:
The linter should be able to identify the versions block regardless of its position in the snapshot file relative to other tests, OR nf-test should respect the source file order when updating snapshots.
Bug 2: Stale Snapshot Persistence (Silent Passing)
Additionally, I observed potentially suboptimal behavior regarding outdated version files.
Observation:
If a snapshot previously contained a versions file check, and the test is subsequently modified to remove that check (or the file is no longer generated), the old entry remains in the *.snap file. nf-test passes successfully because it ignores the extra data in the snapshot, effectively skipping verification for that section.
Resolution:
I had to manually delete the .snap file and regenerate it to clear the stale version entries.
Command used and terminal output
System information
nf-core, version 3.5.1
nf-test 0.9.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the two cases using main.nf.test and main.nf.test.snap, then trace snapshot generation and the test_snap_versions lint check. Done means versions are found regardless of snapshot key order, and removed checks no longer leave stale snapshot data; add regression coverage for both behaviors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100