apple / apple/foundationdb

When all storage replicas of a shard are unavailable, shall we throttle the write traffic to the shard?

Open
#3,613 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

When all storage replicas of a shard are unavailable, read to the shard will be unavailable, which is correct behavior.

The current implementation will still accept write transactions until these transactions fill up tLogs' disks, since those unavailable SSes will not pop data.

Since the cluster is already in a bad state when all replicas become unavailable, it may be a better idea to throttle the write traffic to the shard: (1) if the storage replicas are temporarily unavailable (say 10min), we still accept writes; (2) if it's permanently unavailable, we should reject writes to the shard before the cluster gets worse.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Begin by examining how shard storage-replica availability and tLogs disk consumption are handled, then define the temporary-versus-permanent unavailability policy; done requires an agreed design and its implementation, rather than the question currently specifying a concrete change.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.