apache / apache/lucenenet

[Slow] attribute runs the code in BeforeClass() and AfterClass() even when the `tests:slow` property is set to `false`

Open
#739 0 comments 0 reactions 0 assignees View on GitHub
is:bug pri:normal testability up-for-grabs
Dominant language
C#
Stars
2.4k
Forks
658
Avg merge
3d 5h
Merged PRs (30d)
9

Description

When the `SlowAttribute` in `Lucene.Net.TestFramework` is placed on a `LuceneTestCase`-derived class, it runs the code in the `BeforeClass()` and `AfterClass()` methods (which are decorated with `[OneTimeSetUp]` and `[OneTimeTearDown]`) even when the system property `tests:slow` is set to `false`. This causes these disabled tests to run code, taking up extra testing time. Since the purpose of these settings is to optimize the test run, this undesired behavior is a bug.

I suspect (but haven't confirmed) that the following other attributes also have a similar issue, since they all follow the same basic design.

- `AwaitsFixAttribute`
- `NightlyAttribute`
- `WeeklyAttribute`

Ideally, these would completely suppress any actions from occurring when applied at the class level when their associated [system property](https://github.com/apache/lucenenet/issues/307) is disabled. System properties are statically loaded by the test framework in [LuceneTestCase](https://github.com/apache/lucenenet/blob/c076e40b14d4c20e6fdfee4e28d0b3332cf6d0ce/.github/workflows/Lucene-Net-Tests-AllProjects.yml#L130).

> Note: There is some better info [here](https://lucenenet.apache.org/docs/4.8.0-beta00016/api/test-framework/Lucene.Net.Util.LuceneTestCase.html#randomized-execution-and-test-facilities) on how to specify system properties and command line example [here](https://github.com/apache/lucenenet/blob/c076e40b14d4c20e6fdfee4e28d0b3332cf6d0ce/.github/workflows/Lucene-Net-Tests-AllProjects.yml#L130). No general docs for this yet.

These attributes use NUnit's extensibility through attribute interfaces `IApplyToTest`, `IApplyToContext`, and `IWrapTestMethod` to function. NUnit scans for attributes that implement these interfaces in order to execute them.

Ideally, we find a way to suppress the `[OneTimeSetUp]` and `[OneTimeTearDown]` calls somehow. But these execution calls are owned by NUnit.

Failing that we may be able implement our own attributes with the same names that implement the same interfaces. When these attributes are placed as nested classes of `LuceneTestCase`, they effectively "override" the attributes in the `NUnit.Framework` namespace for `LuceneTestCase` or any class that inherits it.

> Note: We use this approach already for our custom [`[TestFixtureAttribute]`](https://github.com/apache/lucenenet/blob/c076e40b14d4c20e6fdfee4e28d0b3332cf6d0ce/src/Lucene.Net.TestFramework/Support/Util/LuceneTestCase.TestFixtureAttribute.cs#L41). There may be some way to use that custom attribute to fix this where we have direct control over the code. It looks like we are already intercepting these calls [here](https://github.com/apache/lucenenet/blob/c076e40b14d4c20e6fdfee4e28d0b3332cf6d0ce/src/Lucene.Net.TestFramework/Support/Util/LuceneTestCase.SetUpFixture.cs#L70-L92), but I am not sure whether that gives us ownership over all of them.

There is some [custom attribute documentation](https://docs.nunit.org/articles/nunit/extending-nunit/Custom-Attributes.html), but it doesn't actually include `IWrapTestMethod` or [other interfaces that are available in the source code](https://github.com/nunit/nunit/tree/b16d7540ea424d147847ce7af2a576e98a411209/src/NUnitFramework/framework/Interfaces).

Contributor guide

Open the contributing guide

Research direction

Start with SlowAttribute and the custom TestFixtureAttribute in src/Lucene.Net.TestFramework/Support/Util/LuceneTestCase.TestFixtureAttribute.cs, then inspect the interception logic in LuceneTestCase.SetUpFixture.cs and NUnit's IApplyToTest, IApplyToContext, and IWrapTestMethod interfaces. Run the relevant tests with tests:slow=false and verify that class-level BeforeClass and AfterClass actions are suppressed, while checking the related attributes for the same behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.