rust-lang / rust-lang/rustfmt

Refactor system/idempotency tests to use a custom test harness

Open
#4,340 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

P-low
Dominant language
Rust
Stars
7k
Forks
1.1k
Avg merge
2d 13h
Merged PRs (30d)
24

Description

The large amount of system and idempotency tests rustfmt has is great, but each of these test "buckets" run all together as a unit test. As of 3c245a7c3305602ea42b7a3187f5858121c7866b this means we have one unit tests that runs 460 system tests and another that runs 612 idempotency tests. This means there is no convenient way to filter for a single system/idempotency test, making devel of a single feature a bit cumbersome; either

  • you have to wait for all system or idempotency tests to run, or
  • you run an ad-hoc test on the rustfmt binary, which loses the usefulness of cargo test for local devel

My proposal is to refactor the system/idempotency test runner to use a custom test harness (like libtest-mimic). This would allow us to register individual system/idempotency tests and provide compat with all arguments to cargo test, like test filters.

I did this about a month ago for one of my personal projects (https://github.com/yslide/slide/blob/master/slide/src/test/mod.rs), and it's not very difficult. For rustfmt I think it would be as simple as changing the test interface, since the test infra is already set up.

This is low priority.

cc @topecongiro @calebcartwright

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading rustfmt's existing system and idempotency test runner and compare its interface with the libtest-mimic example linked in the issue. Check how the runner is invoked through cargo test and how individual tests are currently grouped. Done means individual system and idempotency tests are registered and cargo test filters and arguments work for them.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing-qa, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.