Lucene KNN token vectors demo
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
I am trying to use a larger vector dictionary with the demo code. I have tried all the files here https://nlp.stanford.edu/projects/glove/ and every one throws this...
`inputs are added out of order lastInput=[74 68 65] vs input=[2e]
java.lang.AssertionError: inputs are added out of order lastInput=[74 68 65] vs input=[2e]
at org.apache.lucene.util.fst.FSTCompiler.add(FSTCompiler.java:478)
at com.trevr.service.lucene.knn.KnnVectorDict$Builder.addOneLine(KnnVectorDict.java:222)`
Contributor guide
Research direction
Reproduce the failure with the Stanford GloVe files and trace the input-order assumption from com.trevr.service.lucene.knn.KnnVectorDict$Builder.addOneLine at line 222 into org.apache.lucene.util.fst.FSTCompiler.add at line 478. Determine why the larger dictionaries violate the expected ordering and make the demo handle the documented dictionary inputs without the assertion failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100