apache / apache/kvrocks

Benchmark for vector indexing and searching

Open
#2,481 4 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

We can try to use this repo: https://github.com/qdrant/vector-db-benchmark

After some simple patching:
```patch
diff --git a/engine/clients/redis/configure.py b/engine/clients/redis/configure.py
index a5e6fe8..b2b8af0 100644
--- a/engine/clients/redis/configure.py
+++ b/engine/clients/redis/configure.py
@@ -7,6 +7,7 @@ from redis.commands.search.field import (
TextField,
VectorField,
)
+from redis.commands.search.indexDefinition import IndexDefinition, IndexType

from benchmark.dataset import Dataset
from engine.base_client.configure import BaseConfigurator
@@ -81,7 +82,7 @@ class RedisConfigurator(BaseConfigurator):
name="vector",
algorithm="HNSW",
attributes={
- "TYPE": "FLOAT32",
+ "TYPE": "FLOAT64",
"DIM": dataset.config.vector_size,
"DISTANCE_METRIC": self.DISTANCE_MAPPING[dataset.config.distance],
**self.collection_params.get("hnsw_config", {}),
@@ -97,8 +98,13 @@ class RedisConfigurator(BaseConfigurator):
]
for conn in conns:
search_namespace = conn.ft()
+ idef = IndexDefinition(
+ prefix=[''],
+ index_type=IndexType.HASH
+ )
+ idef.args = idef.args[:-2]
try:
- search_namespace.create_index(fields=index_fields)
+ search_namespace.create_index(fields=index_fields, definition=idef)
except redis.ResponseError as e:
if "Index already exists" not in str(e):
raise e
diff --git a/engine/clients/redis/upload.py b/engine/clients/redis/upload.py
index cd4b888..0e29533 100644
--- a/engine/clients/redis/upload.py
+++ b/engine/clients/redis/upload.py
@@ -57,7 +57,7 @@ class RedisUploader(BaseUploader):
cls.client.hset(
str(idx),
mapping={
- "vector": np.array(vec).astype(np.float32).tobytes(),
+ "vector": np.array(vec).astype(np.float64).tobytes(),
**payload,
**geopoints,
},

```

We can start a kvrocks instance and run this benchmark:
```bash
python3 run.py --engines redis-default --datasets 'glove-25-angular'
```

cc @Beihao-Zhou

### Solution

Currently we'll get some coredumps in the vector indexing phase, e.g.:
```
E20240809 14:55:31.865105 134 signal_util.h:36] Stack trace (most recent call first):
#0 0x0000555f0502cac9 in SegvHandler at /usr/bin/kvrocks
#1 0x00007f2e599c004f in __sigaction at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f2e59a0ee2c in pthread_key_delete at /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f2e599bffb1 in gsignal at /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f2e599aa471 in abort at /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000555f04f77917 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] at vterminate.o
#6 0x0000555f05b16c09 in __cxxabiv1::__terminate(void (*)()) at /usr/bin/kvrocks
#7 0x0000555f05b16c74 in std::terminate() at /usr/bin/kvrocks
#8 0x0000555f05b16dc7 in __cxa_throw at /usr/bin/kvrocks
#9 0x0000555f04f7908d in std::__throw_bad_alloc() at /usr/bin/kvrocks
#10 0x0000555f04f77509 in handleOOM(unsigned long, bool) [clone .cold] at /kvrocks/build/_deps/jemalloc-src/src/jemalloc_cpp.cpp:90
#11 0x0000555f051d6e4a in redis::VectorItem::VectorItem(redis::VectorItem const&) at /usr/bin/kvrocks
#12 0x0000555f051e561a in redis::IndexUpdater::UpdateHnswVectorIndex(std::basic_string_view >, kqir::Value const&, kqir::Value const&, redis::SearchKey const&, redis::HnswVectorFieldMetadata*) const at /usr/bin/kvrocks
#13 0x0000555f051ec7be in redis::IndexUpdater::Update(std::map, std::allocator >, kqir::Value, std::less, std::allocator > >, std::allocator, std::allocator > const, kqir::Value> > > const&, std::basic_string_view >) const at /usr/bin/kvrocks
#14 0x0000555f05228549 in redis::Connection::ExecuteCommands(std::deque, std::allocator >, std::allocator, std::allocator > > >, std::allocator, std::allocator >, std::allocator, std::allocator > > > > >*) at
/usr/bin/kvrocks
#15 0x0000555f0505db30 in EvbufCallbackBase::readCB(bufferevent*, void*) at /usr/bin/kvrocks
#16 0x0000555f05978a0d in bufferevent_run_deferred_callbacks_unlocked at bufferevent.c
#17 0x0000555f059808a3 in event_process_active_single_queue at event.c
#18 0x0000555f05980f3e in event_process_active at event.c
#19 0x0000555f05982cf8 in event_base_loop.constprop.0 at event.c
#20 0x0000555f0523d46d in std::thread::_State_impl(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() at /usr/bin/kvrocks
#21 0x0000555f05b96312 in execute_native_thread_routine at thread.o
#22 0x00007f2e59a0d133 in pthread_condattr_setpshared at /lib/x86_64-linux-gnu/libc.so.6
#23 0x00007f2e59a8ca3f in __clone at /lib/x86_64-linux-gnu/libc.so.6
```

Maybe we can try to solve it first.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproduced command in the qdrant/vector-db-benchmark repository, then review engine/clients/redis/configure.py and engine/clients/redis/upload.py. Trace the coredump through redis::IndexUpdater::UpdateHnswVectorIndex from the provided stack trace. Done means the glove-25-angular benchmark completes without coredumps during vector indexing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, redis
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.