Investigate indexing slowdown on NYC taxis on 2024-01-18
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
I was looking at the [NYC taxis benchmarks](https://home.apache.org/~mikemccand/lucenebench/sparseResults.html#index_throughput), where we saw a significant indexing rate regression on 2024-01-18.
It's interesting that the dense case got (much) slower while the sparse sorted case got a bit faster and the unsorted sparse case remains similar.
The [diff](https://github.com/apache/lucene/compare/09837bae7378e541813e9a0a226a3107d11c42f4...c661c88fc811d3d5820abe601ed2ed8aa2938514) with the previous data point has several changes that could affect the indexing rate. We need to find which one.
Contributor guide
Research direction
Start with the NYC taxis indexing benchmarks and compare the linked diff between the two data points. Trace the changes that could affect dense, sparse sorted, and unsorted sparse indexing, then identify which change explains the regression and validate it against the benchmark results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100