elsa-workflows / elsa-workflows/elsa-core
Spike: Evaluate TUnit and Microsoft.Testing.Platform for Elsa's test suite
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Objective
Determine whether migrating Elsa's tests from xUnit 2/VSTest to TUnit/Microsoft.Testing.Platform (MTP) provides enough measured CI or developer-loop benefit to justify the migration cost, without changing test semantics, coverage gates, diagnostics, or reliability.
Compare TUnit/MTP with both the current stack and the xUnit v3/MTP migration proposed by #8050. The result should be a recommendation and a staged implementation estimate, not a production migration in this issue.
## Context
Repository inventory was refreshed against [`upstream/main` at `37b1a453`](https://github.com/elsa-workflows/elsa-core/commit/37b1a453169201e577ef6db0bfd0348b08070211).
Elsa currently has 55 xUnit test projects: 37 unit, 17 integration, and 1 component project. The package workflow builds and tests 54 unit/integration projects sequentially in separate processes, while component tests run in a separate job.
Across 15 recent successful `main` runs, the approximate medians were:
| Stage | Median |
| --- | ---: |
| Unit/integration job | 11m 44s |
| `Run tests with coverage` step | 10m 31s |
| Component job | 5m 45s |
| Complete `Packages` workflow | 18m 15s |
TUnit offers compile-time discovery, native MTP integration, and method-level parallel execution by default. Elsa's elapsed time also includes compilation, coverage instrumentation, application-host startup, databases, containers, and workflow execution, so synthetic framework benchmark ratios should not be applied directly to the complete CI workflow.
Preliminary analysis predicts that a semantics-preserving framework change with the current CI topology may range from a small regression to about a 5% improvement, with 0–3% being the most likely result. A 5–15% unit/integration-job improvement may be possible only after separately validating native execution and additional concurrency. Larger gains would more likely come from CI orchestration or sharding and must not be attributed to TUnit alone.
The migration surface is material: more than 600 test source files, custom xUnit discovery, shared fixtures, output helpers, and public `Elsa.Testing.Shared*` APIs are involved.
## Bounded scope
- Compare three configurations on the same CI runner:
1. current xUnit 2 + VSTest;
2. xUnit v3 + MTP, as proposed by #8050;
3. TUnit + MTP, initially retaining `xunit.assert` so runner and lifecycle effects are not mixed with an assertion rewrite.
- Start with one representative fixture-light unit project and one integration-heavy project.
- Use the component project as a fixture-lifetime, serialization, reporting, and diagnostics compatibility check rather than as an initial speed target.
- Keep effective test and project concurrency equivalent for the first comparison.
- Measure clean build, incremental build, no-build execution, production coverage-enabled execution, peak memory, discovered/pass/fail/skip counts, median, p95, and repeated-run flakiness.
- Verify compatibility for GitHub annotations, TRX, hang diagnostics, coverage formats and thresholds, IDE discovery, and public shared test helpers.
- Estimate the manual conversion surface and propose focused implementation PRs if migration is recommended.
## Non-goals
- Repository-wide production migration.
- Rewriting all xUnit assertions.
- Native AOT publishing.
- Unconstrained TUnit parallelism.
- CI sharding or project-level concurrency in the framework-only baseline.
- Fixing existing flaky tests as part of the spike.
- Combining runner, framework, coverage, and parallelism changes into one benchmark result.
## Evidence to collect
### Performance
- Exact commands, package versions, runner image, and hardware allocation.
- Raw timings plus median and p95 from at least five paired CI runs.
- Clean and incremental build time.
- No-build execution through the intended production entry point.
- Production coverage-enabled execution, reported separately from coverage-disabled execution.
- Peak memory and generated report/artifact sizes.
- Attribution of any improvement to MTP, TUnit, or changed concurrency.
### Correctness and reliability
- Exact discovered, passed, failed, and skipped counts, including skip reasons.
- Provider-gated `ConformanceFact`/`ConformanceTheory` behavior.
- Collection, fixture, disposal, and component-test serialization semantics.
- At least 10 consecutive runs of each pilot, following Elsa's testing guideline, with no material flakiness regression.
- Explicit constraints for process-global state, databases, brokers, ports, and shared component fixtures before enabling more parallelism.
### Tooling and compatibility
- GitHub annotations and TRX output.
- Hang detection and dump collection equivalent to the current VSTest configuration.
- Coverage exclusions, Cobertura/report paths, thresholds, and CI failure behavior established by #6950 and #7049.
- Visual Studio, Rider, and VS Code support and minimum versions.
- Compatibility plan for public `Elsa.Testing.Shared*` APIs that expose xUnit types.
- Migration plan for the custom xUnit conformance discoverers.
## Completion criteria
- [ ] A/B/C results are published with commands, versions, raw timings, median, p95, memory, coverage mode, and exact test counts.
- [ ] MTP, framework, coverage, and additional-parallelism effects are reported separately.
- [ ] There is no unexplained discovery/pass/skip delta.
- [ ] Provider skips, fixture lifetimes, serialization, disposal, and nonparallel constraints have a documented TUnit mapping.
- [ ] Repeated pilot runs show no material flakiness regression.
- [ ] CI reporting, TRX, hang diagnostics, coverage outputs, and threshold enforcement have demonstrated parity.
- [ ] Public helper-package and IDE compatibility are documented.
- [ ] Clean/incremental build and local developer-loop effects are included, not only test execution time.
- [ ] The spike recommends one of: TUnit/MTP, xUnit v3/MTP, current stack plus CI optimization, or no change.
- [ ] If migration is recommended, the work is split into small single-concern PRs targeting `main`, with tests and documentation updated as required by `CONTRIBUTING.md`.
- [ ] No production behavior is changed by the spike.
For a performance-only justification, use at least a 10% median reduction in the complete unit/integration job as a suggested decision gate. A smaller result needs a separate, explicit maintainability or developer-experience benefit.
## Related issues and prior work
- #8050 — **open and triaged:** proposes migrating the deprecated xUnit v2 dependencies in `Elsa.Testing.Shared` to xUnit v3; this is the direct alternative and control for the spike.
- #5255 and #5261 — **closed/merged:** established Elsa's extended integration and component-testing infrastructure, whose unit/integration/component boundaries must be preserved.
- #7957 and #7965 — **closed/merged:** document and fix an xUnit parallel-class race involving process-global state; equivalent isolation and parallelism constraints must be verified under TUnit.
- #7404 — **open:** tracks a CI-sensitive flaky component test and should be included in before/after reliability validation.
- #6950 and #7049 — **merged:** established the current Coverlet thresholds, formats, artifacts, and CI coverage gates that an MTP migration must preserve.
Repository issue and PR searches found no existing TUnit or Microsoft.Testing.Platform tracker item. #8050 is the only direct test-framework migration issue found.
## References
- [Elsa contribution guidelines](https://github.com/elsa-workflows/elsa-core/blob/main/CONTRIBUTING.md)
- [Elsa testing strategy](https://github.com/elsa-workflows/elsa-core/blob/main/doc/qa/test-guidelines.md)
- [Current package test workflow](https://github.com/elsa-workflows/elsa-core/blob/main/.github/workflows/packages.yml)
- [Representative successful CI run](https://github.com/elsa-workflows/elsa-core/actions/runs/34747031487)
- [TUnit xUnit migration guide](https://tunit.dev/docs/migration/xunit/)
- [TUnit benchmark methodology](https://tunit.dev/docs/benchmarks/methodology/)
- [Independent matched-MTP benchmark](https://www.meziantou.net/benchmarking-dotnet-test-frameworks-xunit-v3-nunit-mstest-and-tunit.htm)
- [Microsoft test-platform comparison](https://learn.microsoft.com/en-us/dotnet/core/testing/test-platforms-overview)
## Agent readiness
**Agent Ready** — objective, comparison matrix, evidence, boundaries, and validation criteria are defined. Implementation remains out of scope until maintainers review the spike result.
Contributor guide
Research direction
Start with .github/workflows/packages.yml and doc/qa/test-guidelines.md, then benchmark the current stack, xUnit v3/MTP, and TUnit/MTP on one fixture-light unit project and one integration-heavy project. Done means publishing paired-run timings, memory, exact test counts, compatibility and reliability results, plus a recommendation without changing production behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions
- Domain
- ci-cd, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100