dragonflydb / dragonflydb/dragonfly

Background vector indexing correctness

Open
#6,960 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
31.6k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

1. Previosuly all indexing was synchronous under a transaction lock.

```c++
void FtCreate() {
Hop({
for (key : table->traverse())
Add(key)
})
}
```

2. I introduced IndexBuilder https://github.com/dragonflydb/dragonfly/pull/6504 that allows indexing in the background by iterating with Traverse in a separate fiber. It doesn't use transactional locking and relies on the fact that adding to the _classic_ index is a non-suspendable operation

```c++

Fiber({
for (key : table->traverse()) {
Add(key)
ThisFiber::Sleep(x ms); // keeps dragonfly operational even when building a many gigabyte index
}
);
```

3. However, access to a global vector index requires locking a mutex, breaking the assumption of (2).
Currently, the global vector index is a regular std::mutex suspending full thread execution, making this operation safe.

This issue prevents us from adopting a fiber-aware mutex for global hnsw indices

Suggested workarounds:
1. Use a "quiescence" concept where a given "period" is safe for performing modifications, during which indexing can't run. An instance will juggle between those two modes

Contributor guide

Open the contributing guide

Research direction

Start by reading IndexBuilder in PR 6504 and tracing how background traversal interacts with the global vector index and its std::mutex. Compare that behavior with the fiber-aware mutex requirement and evaluate the proposed quiescence workaround. Done means background vector indexing remains safe without blocking full thread execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.