Investigate/Fix NPQ sizing to account for frozen points.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.9k
- Forks
- 454
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 35
Description
Summarized from the discussion in #923.
It seems like the NPQ is configured incorrectly. We need the queue to always be K + num_start_points large because otherwise we may have start points consuming queue slots. That appears to be what happens. There are K results, one of them is the start point, which is filtered out, then the result set returned is only 9 long.
When I (@metajack) originally wrote the filtering code, the NPQ was correctly sized. So somewhere in the last year this has regressed.
Separately, it seems we have a footgun in that it's very easy to forget to add start point filtering to your post processors; i'm not sure what to do about that.
_Originally posted by @metajack in https://github.com/microsoft/DiskANN/pull/923#discussion_r3054749458_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the discussion in pull request #923, then trace where NPQ sizing and start-point filtering are implemented. Verify the queue reserves K plus num_start_points slots and that the filtered result set returns the expected number of results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100