apple / apple/foundationdb

Performant way to extract locality information and shard boundaries

Open
#243 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Currently the only way to get the _full_ picture of where shards are located is to get the shards (that's relatively fast, a single iteration through a system key space) and then ask for the storage servers for each of those shard *one-by-one*. It doesn't help that asking for IP addresses must be throttled so that you don't cause the responsible stateless processes to become pegged. There should be an easier way to accomplish this.

We need this API mainly to build coprocessors that live on the same machine as the DB and needing to figure out what data is local.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the shard, storage-server, locality, and coprocessor APIs, then trace the current one-by-one storage-server lookup and its IP-address throttling. Done means a documented API can return shard locality and boundaries efficiently enough for coprocessors to identify data on their machine.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.