linkedin / linkedin/ambry

Limit number of partitions that data chunks of a "large" blob can reside on

Open
#890 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
1.8k
Forks
298
Avg merge
21h 43m
Merged PRs (30d)
9

Description

Currently Ambry sprays the data chunks of a given "large" blob across all writable partitions (i.e. any partition is eligible to receive a data chunk). This can create some problems
1. If one chunk is lost, the whole blob is considered lost. Spraying it across multiple partitions spread over multiple nodes increases the area of failure
2. One disk being slow/unavailable can have a multiplying effect if it stores a single chunk of many large blobs

In order to restrict these effects, it may be useful to limit the number of partitions the data chunks of a particular "large" blob can reside on. At current time, GET operations request 4 chunks in parallel and maintain the number until the blob is currently served. It may be enough to have 4 * x partitions receive the data chunks (4,8,12 etc) to serve any parallelism requirements.

The partition picking can be random to start with and can be made more intelligent based on requirements (which need to be evaluated)
1. Pick from distinct disks and nodes
2. Pick from distinct disks but limit number of nodes across which these disks are spread (to lessen impact of node failures)
3. ...
Each of the options comes with its own upsides and downsides.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how large-blob data chunks are assigned to writable partitions and how GET operations maintain four chunks in parallel. Evaluate the proposed partition limits and placement options against disk, node, and failure considerations. Done would require a decided design and an implemented, tested restriction on the partitions eligible for each large blob.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.