apple / apple/foundationdb

Performant restore: senders should back off when receivers are stalling due to low free memory

Open
#3,093 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The performant restore currently has the low memory protection on each restore role that will stop any work that is not the current version batch when the role is about to run out of memory.

The sender (i.e., the restore loaders in the scenario of sending mutations from loaders to appliers, for example) simply wait and retry "without extra backoff".

In a real cluster, a role (say applier) that is throttled due to low memory may keep receiving requests that are not going to be processed. This prevents the role from processing the current version batch. When a role does not have enough CPU to process the more important requests, it may be marked as unavailable or connection to it may be marked as unreliable. This can make restore from making progress.

- [ ] We should add the backoff mechanism to avoid such overwhelming requests.
- [ ] Can we simulate the situation and reproduce the problem in simulation?

Note: The problem did not surface in simulation test. It only happened when restore roles have memory pressure.

Contributor guide

Open the contributing guide

Research direction

Start with performant restore’s restore loaders and appliers, focusing on the sender retry path and the existing low-memory protection. Reproduce memory pressure in simulation if possible, then verify that senders back off while receivers are stalled and that the current version batch can make progress.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.