meilisearch / meilisearch/arroy

Optimize deletion by taking a transaction of the previous version of the database

Open
#95 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

breaking enhancement performance
Dominant language
Rust
Stars
311
Forks
17
PR merge metrics
No merged PRs in 30d

Description

Currently, the deletion in arroy runs in O(n) on the number of three nodes which is super slow.
The reason is that when we delete or replace an item, we cannot access the previous version of it, so we don't know where we'll find it, and we have to go through the whole tree.

We could maybe try to implement a slightly better version with https://github.com/meilisearch/arroy/issues/88, but it doesn't seem trivial and won't fix the issue for the updates of item.
Technically, even though it’ll be way faster, it's still O(n) on all the tree nodes, which won't scale forever.

The solution

By taking a read transaction of the previous version of the database, we would be able to read the previous version of the vector under the specified item ID and find it directly in the tree in O(log2(n)).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing arroy’s deletion and replacement paths and the database transaction API; compare the current full-tree scan with issue #88. Define how a read transaction of the previous database version supplies the prior vector, then validate deletion and replacement behavior and the intended complexity.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.