Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka

Add direct memory (off-heap) chunk cache support

未关闭
#830 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
239
派生
59
PR 合并指标
30 天内没有已合并 PR

描述

# What is currently missing?

There are currently 2 chunk cache implementations:

1. **`MemoryChunkCache`** — Stores chunks as `byte[]` on the JVM heap. The cache occupies a large amount of heap memory, and frequent allocation/deallocation of cached data
on the heap increases GC pressure.
2. **`DiskChunkCache`** — Stores chunks as files on disk. Consumes disk space and disk I/O, and reading/writing files has higher latency compared to memory access.

Looking for a cache implementation that avoids both heap memory occupation and disk I/O, with lower GC pressure and lower latency.

# How could this be improved?

Add a `DirectMemoryChunkCache` that stores cached chunks in off-heap (direct) memory using `ByteBuffer.allocateDirect()`. Use a `DirectByteBufferPool` to manage off-heap buffer allocation and reuse. This avoids heap allocation and disk I/O.

Goals:
1. Compared to `MemoryChunkCache`: avoid occupying large amounts of heap memory, eliminate allocation/deallocation on the heap, and reduce GC pressure.
2. Compared to `DiskChunkCache`: avoid consuming disk space and disk I/O, and provide lower latency than reading/writing files.

# Is this a feature you would work on yourself?

* [x] I plan to open a pull request for this feature

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。