apache / apache/lucene

Investigate indexing slowdown on NYC taxis on 2024-01-18

Open
#13,317 0 comments 0 reactions 0 assignees View on GitHub
type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

I was looking at the [NYC taxis benchmarks](https://home.apache.org/~mikemccand/lucenebench/sparseResults.html#index_throughput), where we saw a significant indexing rate regression on 2024-01-18.

It's interesting that the dense case got (much) slower while the sparse sorted case got a bit faster and the unsorted sparse case remains similar.

The [diff](https://github.com/apache/lucene/compare/09837bae7378e541813e9a0a226a3107d11c42f4...c661c88fc811d3d5820abe601ed2ed8aa2938514) with the previous data point has several changes that could affect the indexing rate. We need to find which one.

Contributor guide

Open the contributing guide

Research direction

Start with the NYC taxis indexing benchmarks and compare the linked diff between the two data points. Trace the changes that could affect dense, sparse sorted, and unsorted sparse indexing, then identify which change explains the regression and validate it against the benchmark results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.