Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka
Add direct memory (off-heap) chunk cache support
- Lenguaje dominante
- Java
- Estrellas
- 239
- Forks
- 59
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
# What is currently missing?
There are currently 2 chunk cache implementations:
1. **`MemoryChunkCache`** — Stores chunks as `byte[]` on the JVM heap. The cache occupies a large amount of heap memory, and frequent allocation/deallocation of cached data
on the heap increases GC pressure.
2. **`DiskChunkCache`** — Stores chunks as files on disk. Consumes disk space and disk I/O, and reading/writing files has higher latency compared to memory access.
Looking for a cache implementation that avoids both heap memory occupation and disk I/O, with lower GC pressure and lower latency.
# How could this be improved?
Add a `DirectMemoryChunkCache` that stores cached chunks in off-heap (direct) memory using `ByteBuffer.allocateDirect()`. Use a `DirectByteBufferPool` to manage off-heap buffer allocation and reuse. This avoids heap allocation and disk I/O.
Goals:
1. Compared to `MemoryChunkCache`: avoid occupying large amounts of heap memory, eliminate allocation/deallocation on the heap, and reduce GC pressure.
2. Compared to `DiskChunkCache`: avoid consuming disk space and disk I/O, and provide lower latency than reading/writing files.
# Is this a feature you would work on yourself?
* [x] I plan to open a pull request for this feature
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.