apple / apple/foundationdb

Master proxies can get stuck with high commit latencies

Open
#1,694 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The proxy commit path has protections in place against having more than 5 million versions in flight, to prevent storage servers from not being able to rollback to a recovery version.

If this protection is triggered because of problems with the transaction logs, and the proxy has 8 million versions from the master which is wants to commit but only the first 5M version are in flight, then even if the problem on the transaction logs is resolved the proxy will continue to have high latencies.

This is because new batches of commits can come in at the same rate the logs are completing them, meaning that every batch is waiting 3 seconds before they are submitted to the tlogs.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing the proxy commit path and how transaction-log completion controls batches after the five-million-version limit is reached. Done means that, once transaction-log problems are resolved, commit batches do not continue waiting about three seconds and proxy commit latency returns to normal.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.