apache / apache/lucene

Use RandomMockMergePolicy more often [LUCENE-9469]

Open
#10,508 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:test
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

During the work on #10005 we had some discussions about testability of the feature and how well we can guarantee that it's tested. One argument was that we can rely to a certain extend on the randomization and enabling these features by swapping in RandomMockMergePolicy.
I ran a test which throws an assertion error every time the feature is explicitly used which required MockRandomMergePolicy to be used. Unfortunately I had two entire test runs without any failure except of the tests that explicitly enabled it ie. the ones that I wrote for the feature.
I think we are not using this MP often enough in our tests and I want to propose to use it way more frequently. It's certainly not a replacement for dedicated unit tests, I wrote a dedicated one for every random failure I found which should be common practice but it would be great to increase coverage by swapping in MockRandomMergePolicy more often. Maybe something as simple as this would do it:

```
--- a/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
`@@` -1059,7 +1059,7 `@@` public abstract class LuceneTestCase extends Assert {
}

public static MergePolicy newMergePolicy(Random r, boolean includeMockMP) {
- if (includeMockMP && rarely(r)) {
+ if (includeMockMP && r.nextBoolean()) {
return new MockRandomMergePolicy(r);
} else if (r.nextBoolean()) {
return newTieredMergePolicy(r);
```

---
Migrated from [LUCENE-9469](https://issues.apache.org/jira/browse/LUCENE-9469) by Simon Willnauer (@s1monw)

Contributor guide

Open the contributing guide

Research direction

Start in lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java, at newMergePolicy and its MockRandomMergePolicy selection. Trace the tests that use the includeMockMP path and run the relevant test suite to evaluate randomized coverage. Done means MockRandomMergePolicy is exercised more frequently without replacing dedicated feature tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.