apache / apache/lucene

Improve Norms merging APIs [LUCENE-3787]

Open
#4,860 0 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Spinoff from #3706, since it took us a fair amount of time to track down,
I think its worth trying to improve the API.

The DocValuesConsumer api's default merge implementation calls canMerge/getDocValuesForMerge/getDocValuesType (protected methods).

but its a little strange how this works:
- all norms implementations must override the default DV implementation, or they might accidentally merge docvalues into their norms!
- preflex-RW only overrides 2 of these... how is its norms merging working... is it?
- its tricky obviously for issues like #3706: as delegating merge() is not very obvious either.

So I think we should look at this, instead of having all NormsWriters override DocValues, redefining these methods in a way thats
not a "is-a" relationship, we could do something else like have split NormsConsumer/DVConsumer apis that share a package-private base class.

This way a norms impl just extends NormsConsumer and there are no traps.

---
Migrated from [LUCENE-3787](https://issues.apache.org/jira/browse/LUCENE-3787) by Robert Muir (@rmuir)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the DocValuesConsumer default merge implementation and its protected canMerge, getDocValuesForMerge, and getDocValuesType methods, then compare how norms implementations and preflex-RW override them. The issue's proposed direction is to separate NormsConsumer and DVConsumer APIs; done should remove the accidental docvalues-merging trap while preserving correct norms merging.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.