apache / apache/lucene

Sorting on FloatFiled or FloatPoint producing erroneous results [LUCENE-10515]

Open
#11,551 0 comments 0 reactions 0 assignees View on GitHub
affects-version:5.3.1 legacy-jira-priority:Major module:core/search type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When we use FloatField for sorting, decimal values are getting rounded off and results is erroneous. Even after upgrading version to 6.0.0 FloatPoint results are also erroneous. We are getting correct results only when we use field as FloatDocValues.

I have attached a sample project which can show case the bug which i encountered [LuenceSortFloatIssue.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/LuenceSortFloatIssue.zip)

I debugged to see what happened internally, below are few things which i encountered.

I used the following values to sort 12.5,15,0,0,12. We can clearly see 12.5 and 12 are saved as 1.7E-44 in queue.
![FloatFieldStored.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/FloatFieldStored.png)

when we use `FloatDocValuesField` to store the value then the sorting is working as expected. we can see values are stored without rounding off in below image
![FloatDocValuesStored.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/FloatDocValuesStored.png)

Only difference i see in the file Lucene50DocValuesProducer . In case of floatPoint numeric entry format is CONST_COMPRESSED and for rest of field types like int point, long point or FloatDocValuesField the entry format is TABLE_COMPRESSED.
![screenshot-1.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/screenshot-1.png)

![Lucene50DocValuesProducer.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/Lucene50DocValuesProducer.png)

---
Migrated from [LUCENE-10515](https://issues.apache.org/jira/browse/LUCENE-10515) by SAILENDRA PAVAN
Attachments: [FloatDocValuesStored.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/FloatDocValuesStored.png), [FloatFieldStored.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/FloatFieldStored.png), [Lucene50DocValuesProducer.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/Lucene50DocValuesProducer.png), [LuenceSortFloatIssue.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/LuenceSortFloatIssue.zip), [screenshot-1.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10515/screenshot-1.png)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the sorting behavior with the attached sample project and the values 12.5, 15, 0, 0, and 12. Then inspect Lucene50DocValuesProducer and the FloatField, FloatPoint, and FloatDocValuesField paths, comparing their numeric entry formats. Done means decimal sorting for FloatField and FloatPoint preserves the expected ordering instead of rounding values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.