emqx / emqx/mria

[RFC] Speed up bootstrap by maintaining a "delta set" table

Open
#145 1 comment 1 reaction 0 assignees View on GitHub
Feature
Dominant language
Erlang
Stars
158
Forks
22
PR merge metrics
No merged PRs in 30d

Description

Currently replicants have to copy the entire contents of the tables when they reconnect, even after a short while. With large enough volume of data it may hinder cluster recovery after disaster or maintenance.

Initially we tried to solve this problem by persisting the transaction log, so the replicants recovering after a reconnect could replay it instead of going through the entire bootstrap procedure.

That approach proved to hurt performance too much to be practical. In addition, flapping client connections can often generate `delete -> add -> delete -> ...` loops in the transaction logs, making them larger than the table itself, making the whole idea of replaying transaction log questionable.

Below I describe an alternative approach. Instead of trying to avoid bootstrap, we could speed it up.

1. For each shard we create so called "delta set", that consists of N `set`-like tables (plain ets or plain rocksdb) that store the following records: `{{Table, Key}, X}` where `X` is a value of a counter.
2. Every M seconds we rotate the tables in the delta set in a ring buffer fashion, contents of the oldest table are dropped. We also increase counter `X` by 1.
3. `mria_rlog_server` process, as it processes intercepted transactions, writes each affected key to the table with the current delta set table. The existing keys are simply overwritten.
4. When a replicant connects to the core, it tells it its current logical timestamp in the hello message
5. The core checks if the timestamp is covered by its delta set.
6. If it is, instead of doing the normal bootstrap server loop (https://github.com/emqx/mria/blob/main/src/mria_bootstrapper.erl#L192) it loops over the delta set keys, and it doesn't send `clear_table` command (https://github.com/emqx/mria/blob/main/src/mria_bootstrapper.erl#L240), so the replicant preserves its local data.
7. If the replicant's data is too old, bootstrap server does the normal loop.

# Pitfalls:

1. Clock skews. Possible solution: make sure to replay much older keys than the replicant's timestamp.
2. Care should be taken to avoid "skipping" over a part of delta set, as tables get rotated while the bootstrap is running. This could be prevented, perhaps, by making sure `X` counter doesn't change during bootstrap, and only increments by 1 while jumping to the next table.
3. When the core node itself restarts, or rlog_server process restarts, it can skip certain keys. In this situation replaying the delta set would lead to inconsistent results, so the best course of action would be to simply drop it. It means any replicant node that connects to a freshly restarted core node has to bootstrap from scratch.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the bootstrap loop in src/mria_bootstrapper.erl around the referenced lines, then trace how mria_rlog_server processes intercepted transactions and how the replicant sends its logical timestamp. Done means the proposed delta-set path is designed with rotation, clock skew, and restart handling, while old or invalid state still uses the normal bootstrap.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.