Automattic / Automattic/harper

Tests in `harper-core/tests/run_tests.rs` are poor at communicating their expectations

Open
#2,625 0 comments 0 reactions 0 assignees View on GitHub
harper-core testing
Dominant language
Rust
Stars
15.4k
Forks
627
Avg merge
1d 15h
Merged PRs (30d)
102

Description

When they fail, the only thing they communicate is the entire list of lints they got, and that the number of lints doesn't match a magic number they expect.

For most of these tests, this is just an inconvenience. They're short and only expect a few lints, so it's not too difficult to look into them and figure out what they expect. However, there is one test that expects specifically 49 lints. If that fails, well... good luck.

Aside from the difficulty in debugging, this is also a weakness in the tests. Since they're only checking the number of lints, not the type, they can pass even if they get incorrect lints.

Contributor guide

Open the contributing guide

Research direction

Start in harper-core/tests/run_tests.rs and inspect the tests that currently compare lint counts, especially the test expecting 49 lints. Run the existing test suite to observe the failure output, then make the expectations identify the expected lint types rather than only a magic count. Done means failures explain which expectations differ and incorrect lint types cannot pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
developer-experience, testing
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.