dragonflydb / dragonflydb/dragonfly
Background vector indexing correctness
- Dominant language
- C++
- Stars
- 31.6k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
1. Previosuly all indexing was synchronous under a transaction lock.
```c++
void FtCreate() {
Hop({
for (key : table->traverse())
Add(key)
})
}
```
2. I introduced IndexBuilder https://github.com/dragonflydb/dragonfly/pull/6504 that allows indexing in the background by iterating with Traverse in a separate fiber. It doesn't use transactional locking and relies on the fact that adding to the _classic_ index is a non-suspendable operation
```c++
Fiber({
for (key : table->traverse()) {
Add(key)
ThisFiber::Sleep(x ms); // keeps dragonfly operational even when building a many gigabyte index
}
);
```
3. However, access to a global vector index requires locking a mutex, breaking the assumption of (2).
Currently, the global vector index is a regular std::mutex suspending full thread execution, making this operation safe.
This issue prevents us from adopting a fiber-aware mutex for global hnsw indices
Suggested workarounds:
1. Use a "quiescence" concept where a given "period" is safe for performing modifications, during which indexing can't run. An instance will juggle between those two modes
Contributor guide
Research direction
Start by reading IndexBuilder in PR 6504 and tracing how background traversal interacts with the global vector index and its std::mutex. Compare that behavior with the fiber-aware mutex requirement and evaluate the proposed quiescence workaround. Done means background vector indexing remains safe without blocking full thread execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100