microsoft / microsoft/testfx

[Feature]: Add differ extensibility for MSTest assertions

Open
#6,771 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/assertion area/mstest
Dominant language
C#
Stars
1k
Forks
312
Avg merge
8h 30m
Merged PRs (30d)
469

Description

Summary

One current issue for assertions is the level of clarity of error messages. There is probably no one solution that fits all scenarios.

Background and Motivation

We should aim for a good experience when a test fails. For that, a degree of extensibility might be best.

Proposed Feature

public sealed class Assert
{
+    public void AddExtension(IAssertDiffExtension assertDiffExtension);
}

+ public interface IAssertDiffExtension
+ {
+     // TODO: Should this take the original objects instead?
+     bool IsSupported(Type typeOfExpected, Type typeOfActual);
+     string Diff(object expected, object actual);
+ }

Multiple extensions can be added during AssemblyInitialize. When an assert fails and wants to diff two objects, we will loop over all registered diff extensions (in reverse order). The first found extension that returns true from IsSupported will be used (which is the last registered because we loop in reverse order - more on the motivation of reverse ordering later in the proposal).

  • This allows differs to be implemented outside of MSTest.TestFramework, as separate packages. There can be some provided by us, and others provided by the community.
    • One example is using DiffPlex here, potentially with different options that users can configure. For example, users who knows they have lots of multi-line string comparisons can install a differ that plays more nice with that, etc.
  • Having the extensibility here allows TestFramework to not require additional dependencies (e.g, DiffPlex). The additional dependency is installed only when user explicitly wants the DiffPlex differ extension.
  • The actual storage of the extensions can be an async local.
    • When installing in AssemblyInitialize, it makes it available to all tests.
    • Two different test classes can be running in parallel, and each test class could install a different differ because of the nature of the tests in those specific test classes.
    • That is why the "last registered" wins rule. If I install a differ in AssemblyInitialize (registered first), I want it to apply it everywhere. If later I installed one in ClassInitialize, I want that to be "more specific" and win.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the proposed Assert.AddExtension and IAssertDiffExtension API in the issue, including the open question about passing original objects. Define the registration order, async-local isolation, supported-type selection, and fallback behavior before locating the relevant implementation and tests. Done means the API and behavior are agreed and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.