microsoft / microsoft/testfx

Support tests sharding in dotnet test

Open
#4,068 7 comments 16 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/mtp
Dominant language
C#
Stars
1k
Forks
312
Avg merge
8h 30m
Merged PRs (30d)
469

Description

### Is your feature request related to a problem? Please describe.
We are trying to reduce the time of our CI by splitting the execution between multiple runners.

### Describe the solution you'd like

Similar to what [we can do in Playwright](https://playwright.dev/docs/test-sharding#sharding-tests-between-multiple-machines), I think that implementing sharding in the `dotnet test` CLI would greatly benefit many projects. One possible API could be based on the previous example:

```bash
# The first runner could execute this, running the first fifth of the tests in MyTests.dll
dotnet test "MyTests.dll" --shard 1/5
# Run the second fifth of the tests in MyTests.dll
dotnet test "MyTests.dll" --shard 2/5
# ...
dotnet test "MyTests.dll" --shard 3/5
dotnet test "MyTests.dll" --shard 4/5
dotnet test "MyTests.dll" --shard 5/5
```

### Additional context
We already tried these two solutions that does not completely satisfy us:

#### Work around the issue by splitting our tests in multiple projects

We can have multiple test projects and configure our CI to split the projects to run between multiple runners. This is not ideal as it forces an architecture upon us, multiplying the projects count for no other purpose than the tests execution.

#### Keep a single test project and split the tests to run manually

We also tried to implement our own "sharding" by:

1. Listing all of the tests with the `dotnet test --list-tests` command;
2. Split the result in shards in a bash script;
3. Have one CI runner per shard.

The return of the `dotnet test --list-tests` command forces us to manipulate it multiple times, removing the headers that cannot be muted and manage the parameterized tests that appears multiple times. Those manipulations are tedious and seem hacky, compared to a native solution provided by the SDK.

----

Opened by @ThomasFerroAgicap in https://github.com/dotnet/sdk/issues/42986#issuecomment-2474730390

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the `dotnet test` CLI discovers and partitions tests, using the issue's `--list-tests` workflow as context. Compare the requested `--shard 1/5` behavior with the existing test-runner architecture, then define completion as reliable shard selection for parameterized tests across multiple runners with coverage in the relevant test suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.