apache / apache/lucenenet

Port RandomizedTesting Test Runner

Open
#264 4 comments 0 reactions 0 assignees View on GitHub
pri:low up-for-grabs
Dominant language
C#
Stars
2.4k
Forks
658
Avg merge
3d 5h
Merged PRs (30d)
9

Description

While the test framework can run with the standard NUnit test runner and we have managed to make it work, it is missing the ability to reproduce a random test failure, which is crucial for debugging some of the more complicated random tests. Lucene uses a custom JUnit test runner called RandomizedTesting to accomplish this.

It also includes some other nice features


  1. Ensure when a test fails, the random seeds are included in error messages and logs

  2. Code analysis to ensure the tests are setup properly

  3. Run tests in a random order to ensure they are not dependent upon each other

Preliminary analysis shows that the API for NUnit allows building custom test runners and it is a close enough match to implement the functionality. Most likely, this will require a custom adapter as well, so the test runner can integrate into Visual Studio and dotnet test/mstest.

Do note that without doing some pretty heavy refactoring on the Codec, Directory, and LuceneTestCase classes, it is not possible to run tests in parallel within the same AppDomain because the codecs use a static variable to turn codec impersonation on/off. For now, it would probably be best to run tests serially.


JIRA link - [LUCENENET-627] created by nightowl888

Contributor guide

Open the contributing guide

Research direction

Start by reviewing NUnit's custom test-runner API and the existing Codec, Directory, and LuceneTestCase classes. Investigate how a custom adapter could integrate with Visual Studio and dotnet test/mstest while preserving serial execution. Done means reproducible random failures, seed reporting, setup analysis, and randomized test ordering work through the supported test entry points.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.