redpanda-data / redpanda-data/redpanda
Enable skipping cloud storage cache for small segments
@abhijat is already working on this.
Since Oct 3, 2023.
- Dominant language
- C++
- Stars
- 12.6k
- Forks
- 792
- PR merge metrics
- No merged PRs in 30d
Description
Each file in the cache path has to be accounted for in the cache trim. This means that when the cache is walked before trim, the metadata for each file is held in memory, and the data structure has to be sorted before trimming.
For certain deployments the cache object count can range into very large numbers. Some segment files (particularly compacted segments after reupload) are so small that they can be served directly from the HTTP stream without writing them to disk.
This makes sure that the cache eviction does not have to work on sorting them by access time. A tunable threshold (something like 1MiB) can be added below which the segment will be served directly from the download stream instead of first writing it to disk.
JIRA Link: CORE-1483
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.