apache / apache/paimon-vector-index

[Feature] Add CI benchmarks to compare vector index PRs with their base

Open
#96 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21
Forks
21
Avg merge
2d 10h
Merged PRs (30d)
10

Description

Vector index changes can affect query throughput or recall without failing correctness tests. Reviewers currently need to run benchmarks manually to compare a PR with its base revision.

Add a lightweight CI benchmark that compares both revisions on the same runner with the same workload and benchmark driver. The report should make recall and query-throughput changes easy to scan, retain raw samples for investigation, and use advisory thresholds rather than fail CI on noisy timing differences.

The initial scope is a small, repeatable warm-cache workload covering IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ and DiskANN. Results should be available in the Actions summary, with an updated PR comment where token permissions allow it.

Implementation: #95.

Contributor guide

Open the contributing guide

Research direction

Read implementation issue #95 first, then locate the existing CI workflow and benchmark driver. Verify a repeatable warm-cache workload covering IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ and DiskANN compares both revisions on the same runner, publishes recall and throughput in the Actions summary, retains raw samples, uses advisory thresholds, and updates the PR comment when permitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd, performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.