apache / apache/lucene

pathological test fixes [LUCENE-9817]

Open
#10,856 8 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

There are now 13,000+ tests in lucene, and if you don't have dozens of cores the situation is slow (around 7 minutes here, with everything tuned as fast as i can get it, running on tmpfs).

It is tricky to keep the situation sustainable: so many tests that usually just take a few seconds but they all add up. To put it in perspective, imagine if all 13000 tests only took 1s each, that's 3.5 hours of cpu time.

From my inspection, there are a few cases of inefficiency:
- tests with bad random parameters: they might normally be semi-well-behaved, but "rarely" take 30 seconds. That's maybe like a 1% chance but keep in mind 1% equates to 130 wild-west tests every run.
- tests spinning up too many threads and indexing too many docs unnecessarily: there might literally be thousands of these, so that's a hard problem to fix... and developers love to use lots of threads and docs in tests.
- tests just being inefficient: stuff like creating indexes in setup/teardown when they have many methods that may not even use them (hey, why did testEqualsHashcode take 30 seconds, what is it doing?)

I only worked on the first case here, if i fixed anything involving the other two, it was just because I noticed them while I was there. I temporarily overrode methods like LuceneTestCase.rarely(), atLeast(), and so on to present more pathological/worst-case conditions and tried to address them all.

So here's a patch to give \~ 80 seconds of cpu-time in tests back. YMMV, maybe it helps you more if you are actually using hard disks and stuff!

Fixing the other issues here will require some more creativity/work, I will followup.

---
Migrated from [LUCENE-9817](https://issues.apache.org/jira/browse/LUCENE-9817) by Robert Muir (@rmuir), updated Mar 16 2021
Attachments: [LUCENE-9817.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9817/LUCENE-9817.patch) (versions: 3)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached LUCENE-9817.patch and the LuceneTestCase.rarely() and atLeast() methods mentioned in the issue. Compare the affected tests under pathological random conditions and verify that the focused changes reduce unnecessary CPU time without weakening coverage; the issue does not name specific test files or define remaining follow-up work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.