zilliztech / zilliztech/VectorDBBench

🚀 Feature Request: Manual Search QPS Control

Open
#566 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
438
Avg merge
1d 4h
Merged PRs (30d)
1

Description

Problem

Streaming tests only measure maximum QPS by running searches as fast as possible. No option exists to test performance under controlled, specific QPS loads.

Current vs Proposed
  1. Current: insert_rate controllable, search rate runs at maximum speed
  2. Proposed: Add optional search_qps parameter for controlled search rates
Use Cases
  1. SLA Testing: "Can DB maintain 95% accuracy at 1000 QPS?"
  2. Resource Planning: "What's latency at target 500 QPS?"
  3. Load Comparison: "DB A vs B performance at same controlled load"
  4. Bottleneck Analysis: "At what QPS does performance degrade?"
Expected Behavior
  1. When search_qps not specified → existing max QPS measurement (backward compatible)
  2. When search_qps specified → test performance at that specific rate
  3. Results show both target QPS and achieved metrics
Questions
  1. Does this feature align with project roadmap?
  2. Any preferred approach or naming conventions?
  3. Should this extend to other test types beyond streaming?
Next Steps

I'm planning to contribute this feature to VectorDBBench. Should I go ahead with the implementation?
Any guidance on preferred approach or design considerations would be appreciated before I start working on it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the streaming benchmark's search loop and the existing insert_rate handling. Determine how an optional search_qps value can preserve maximum-speed behavior when omitted and report both target and achieved rates. Done means controlled-rate streaming tests work without changing existing defaults, with guidance from the unresolved design questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.