apple / apple/foundationdb

Create multiple best locations for each tag in the TagPartitionedLogSystem

Open
#2,510 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

If a transaction commits a large amount of data that all goes in a single shard, all the mutations in that transaction will be sent to the same tlogs.

This causes higher latencies, because only a small percentage of the tlogs is involved in processes that commit.

To avoid this problem, we can create multiple best locations for each tag, and storage servers must merge mutations from all best locations. Now, a proxy can randomly send a mutation to any of the best locations for a given tag, allowing more tlogs to be involved in these commits.

When implementing this feature, we need to make sure that a given mutation is not sent to more than one best location for any tag, so that we avoid storage servers receiving the same mutation multiple times.

Contributor guide

Open the contributing guide

Research direction

Start by locating the TagPartitionedLogSystem implementation and the proxy and storage-server mutation-routing paths. Review how a tag's best location is selected and how mutations are merged, then identify tests for multiple locations and duplicate prevention. Done means mutations can use multiple best locations while each mutation reaches no more than one location per tag.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
database, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.