Performant way to extract locality information and shard boundaries
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Currently the only way to get the _full_ picture of where shards are located is to get the shards (that's relatively fast, a single iteration through a system key space) and then ask for the storage servers for each of those shard *one-by-one*. It doesn't help that asking for IP addresses must be throttled so that you don't cause the responsible stateless processes to become pegged. There should be an easier way to accomplish this.
We need this API mainly to build coprocessors that live on the same machine as the DB and needing to figure out what data is local.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by locating the shard, storage-server, locality, and coprocessor APIs, then trace the current one-by-one storage-server lookup and its IP-address throttling. Done means a documented API can return shard locality and boundaries efficiently enough for coprocessors to identify data on their machine.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100