nf-core / nf-core/tools

test_snap_versions lint failure due to snapshot key ordering mismatch (and stale snapshot persistence)

Open
#3,969 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Description of the bug

Description
I have encountered two related behaviors regarding nf-test snapshots and nf-core modules lint that lead to false failures and stale data management issues.

Bug 1: Snapshot Ordering Mismatch causes Lint Failure
When a module has multiple tests (e.g., a script test and a stub test), the order in which entries appear in the *.nf.test.snap file can drift from the order in main.nf.test. This causes nf-core modules lint to report missing versions, even when they exist.

Steps to Reproduce:
Create a module with two tests in main.nf.test:

Test A: A "real" script test (initially configured without snapshotting output/versions).

Test B: A stub test (configured with assertions/snapshots).

Run nf-test test.

Result: main.nf.test.snap is created containing only Test B data.

Modify Test A to assert/snapshot versions.yml.

Run nf-test test again.

Result: Test A data is appended to the bottom of the existing main.nf.test.snap.

Current File State:

main.nf.test: Order is [Test A, Test B].

main.nf.test.snap: Order is [Test B, Test A].

Run nf-core modules lint.

Observed Behavior:
The linter fails with the error:

test_snap_versions: versions not found in snapshot
This occurs because the linter appears to expect the snapshot keys to appear in a specific order or struggles to parse the versions when the script test appears after the stub test in the snapshot file.

Expected Behavior:
The linter should be able to identify the versions block regardless of its position in the snapshot file relative to other tests, OR nf-test should respect the source file order when updating snapshots.

Bug 2: Stale Snapshot Persistence (Silent Passing)
Additionally, I observed potentially suboptimal behavior regarding outdated version files.

Observation:
If a snapshot previously contained a versions file check, and the test is subsequently modified to remove that check (or the file is no longer generated), the old entry remains in the *.snap file. nf-test passes successfully because it ignores the extra data in the snapshot, effectively skipping verification for that section.

Resolution:
I had to manually delete the .snap file and regenerate it to clear the stale version entries.

Command used and terminal output

System information

nf-core, version 3.5.1
nf-test 0.9.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two cases using main.nf.test and main.nf.test.snap, then trace snapshot generation and the test_snap_versions lint check. Done means versions are found regardless of snapshot key order, and removed checks no longer leave stale snapshot data; add regression coverage for both behaviors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.