redpanda-data / redpanda-data/redpanda

Enable skipping cloud storage cache for small segments

Open
#13,883 0 comments 0 reactions 1 assignee View on GitHub

@abhijat is already working on this.

Since Oct 3, 2023.

area/cloud-storage kind/enhance
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

Each file in the cache path has to be accounted for in the cache trim. This means that when the cache is walked before trim, the metadata for each file is held in memory, and the data structure has to be sorted before trimming.

For certain deployments the cache object count can range into very large numbers. Some segment files (particularly compacted segments after reupload) are so small that they can be served directly from the HTTP stream without writing them to disk.

This makes sure that the cache eviction does not have to work on sorting them by access time. A tunable threshold (something like 1MiB) can be added below which the segment will be served directly from the download stream instead of first writing it to disk.

JIRA Link: CORE-1483

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.