apache / apache/iceberg-rust

[EPIC] Iceberg Cache

Open
#1,226 5 comments 6 reactions 0 assignees View on GitHub
epic
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### What's the feature are you trying to implement?

Cache is an essential component of an Iceberg table, and different types of cache are needed at various levels.

For example, for our table metadata, we will need a `Manifest` cache so that we don't have to read and deserialize the same manifest files repeatedly. For our Parquet files, we will need a `FileMetadata` cache to avoid parsing the metadata from the Parquet files each time. We could even implement a raw data cache to store portions of data files, eliminating the need to download them from S3 again.

As the foundation for various query engines, iceberg-rust should be designed to simplify integration while still allowing each engine to fully optimize performance. This applies whether they are using iceberg-rust on a single machine or within a distributed cluster.

I plan to add a set of cache APIs to meet all those needs. My current plan is:

- `ObjectCache`: an object cache trait that can hold objects like `Manifest` or `FileMetadata`
- `BytesCache`: a bytes cache that can hold row content of files, like `table_metadata.json` files.
- In FileIO Cache like opendal's CacheLayer, but the API is not decided yet.

## Tasks

- ObjectCache
- [x] https://github.com/apache/iceberg-rust/pull/1222
- [ ] https://github.com/apache/iceberg-rust/pull/1225
- [ ] https://github.com/apache/iceberg-rust/pull/1227
- BytesCache
- OpenDAL CacheLayer (TBD)

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ObjectCache work referenced in PRs 1222, 1225, and 1227 to understand the planned cache APIs. Then determine the scope for BytesCache and the still-undecided OpenDAL CacheLayer integration; the epic is complete when the planned cache API work is implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.