apache / apache/lucene

BufferedUpdates is memory inefficient [LUCENE-10217]

Open
#11,254 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I recently got a question from David Turner about why `IndexWriter` was flushing data so frequently despite very small documents. After investigating, we noticed that most of the RAM buffer was actually spent on BufferedUpdates since his test was using `IndexWriter#updateDocument`. This is not surprising given that BufferedUpdates accounts BYTES_PER_DEL_TERM=160 bytes per update, plus the length of the field and the length of the term, so often around 200 bytes only to record the updated term.

As a comparison, Lucene's nightly NYC taxis benchmark only needs 286 bytes per document in the RAM buffer for about 20 fields, (http://people.apache.org/\~mikemccand/lucenebench/sparseResults.html#index_docs_per_mb_ram), or \~15 bytes per field.

Updates are expected to be slower than appending given that they need to look up terms in the dictionary, but I suspect that this memory inefficiency is making updates even slower by forcing Lucene to flush its RAM buffer much more frequently than it has to when purely appending documents.

---
Migrated from [LUCENE-10217](https://issues.apache.org/jira/browse/LUCENE-10217) by Adrien Grand (@jpountz)

Contributor guide

Open the contributing guide

Research direction

Start by tracing IndexWriter#updateDocument into BufferedUpdates, focusing on the BYTES_PER_DEL_TERM accounting described in the issue. Compare its per-update memory use with the NYC taxis benchmark, then establish how a change would preserve update behavior while reducing unnecessary RAM-buffer flushing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.