Performant restore: senders should back off when receivers are stalling due to low free memory
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The performant restore currently has the low memory protection on each restore role that will stop any work that is not the current version batch when the role is about to run out of memory.
The sender (i.e., the restore loaders in the scenario of sending mutations from loaders to appliers, for example) simply wait and retry "without extra backoff".
In a real cluster, a role (say applier) that is throttled due to low memory may keep receiving requests that are not going to be processed. This prevents the role from processing the current version batch. When a role does not have enough CPU to process the more important requests, it may be marked as unavailable or connection to it may be marked as unreliable. This can make restore from making progress.
- [ ] We should add the backoff mechanism to avoid such overwhelming requests.
- [ ] Can we simulate the situation and reproduce the problem in simulation?
Note: The problem did not surface in simulation test. It only happened when restore roles have memory pressure.
Contributor guide
Research direction
Start with performant restore’s restore loaders and appliers, focusing on the sender retry path and the existing low-memory protection. Reproduce memory pressure in simulation if possible, then verify that senders back off while receivers are stalled and that the current version batch can make progress.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100