apache / apache/kvrocks

To enable baseline comparison for performance optimization PRs, kvrocks needs benchmark and latency testing support

Open
#3,408 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

I compared the benchmark testing in Redis and kvrocks, and found that kvrocks lacks benchmark testing and latency testing. Here's the comparison table:

| Test Type | Redis | kvrocks |
|-----------|-------|---------|
| Unit Test | ✅ | ✅ |
| Integration Test | ✅ | ✅ |
| Benchmark Test | ✅ redis-benchmark | ❌ |
| Latency Test | ✅ --latency, --latency-history | ❌ |

This makes performance-related PRs difficult to review, since contributors cannot provide performance comparisons (unless they use these tools to test themselves, but this adds burden to
contributors).

Therefore, I believe kvrocks needs to support running these tests locally and provide a comprehensive test suite, allowing contributors to easily run performance tests within the kvrocks
repository.

### Solution

Support benchmark testing via redis-benchmark, and provide latency testing via redis-cli. Meanwhile, migrate the test suites from Redis (only the parts that kvrocks already supports).

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the Redis benchmark and latency testing tools with the test suites that Kvrocks already supports. Identify how redis-benchmark and redis-cli can run locally, then verify that supported Redis tests cover benchmark and latency scenarios and that contributors can reproduce performance comparisons.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, redis
Domain
databases, performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.