NVIDIA / NVIDIA/cuvs

[FEA] Supporting filters in the Fastener implementation of CAGRA merge

Open
#2,417 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Is your feature request related to a problem? Please describe.
The Fastener CAGRA merge procedure introduced in #2352 provides a faster method for merging CAGRA graphs than the alternate rebuilding approach. However, it does not support the row_filter arg, and the AUTO merge strategy currently falls back to rebuilding if a filter is provided. Compaction is a natural time to remove tombstoned points in a LSM tree, so this is likely a priority for usability in this setting.

Describe the solution you'd like
Adapting Fastener to support functionally point deletion is not trivial for the same reason deleting points in a search graph isn't: removed points can break the connectivity of the graph. However, the merge being done by Fastener is already over disjoint graphs, and the process of adding the scaffold, if minorly adjusted to allow intra-input scaffold edges, would likely repair most minor deletions. Above a certain fraction of points deleted, say 10 or 25%, rebuilding would again be necessary, but this seems like it would be uncommon, especially in large merges where rebuilding would be expensive.

Describe alternatives you've considered
It's possible that no logic needs to change for filters to be supported effectively, and this is largely a benchmarking issue. Regardless, solid benchmarks of post-merge graph quality need to be done before filters are enabled for Fastener.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Fastener CAGRA merge introduced in #2352 and inspect how row_filter and AUTO currently select rebuilding; the issue names no files or tests. Benchmark post-merge graph quality with filters before deciding whether Fastener changes are needed, and validate filtered merges or document rebuilding as the appropriate choice.

Written by the indexing model from the issue text.

Assessment

Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.