BufferedUpdates is memory inefficient [LUCENE-10217]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I recently got a question from David Turner about why `IndexWriter` was flushing data so frequently despite very small documents. After investigating, we noticed that most of the RAM buffer was actually spent on BufferedUpdates since his test was using `IndexWriter#updateDocument`. This is not surprising given that BufferedUpdates accounts BYTES_PER_DEL_TERM=160 bytes per update, plus the length of the field and the length of the term, so often around 200 bytes only to record the updated term.
As a comparison, Lucene's nightly NYC taxis benchmark only needs 286 bytes per document in the RAM buffer for about 20 fields, (http://people.apache.org/\~mikemccand/lucenebench/sparseResults.html#index_docs_per_mb_ram), or \~15 bytes per field.
Updates are expected to be slower than appending given that they need to look up terms in the dictionary, but I suspect that this memory inefficiency is making updates even slower by forcing Lucene to flush its RAM buffer much more frequently than it has to when purely appending documents.
---
Migrated from [LUCENE-10217](https://issues.apache.org/jira/browse/LUCENE-10217) by Adrien Grand (@jpountz)
Contributor guide
Research direction
Start by tracing IndexWriter#updateDocument into BufferedUpdates, focusing on the BYTES_PER_DEL_TERM accounting described in the issue. Compare its per-update memory use with the NYC taxis benchmark, then establish how a change would preserve update behavior while reducing unnecessary RAM-buffer flushing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100