apache / apache/paimon

[Feature] Download remote index files when already built

Open
#8,123 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Now that Paimon enables remote index files for data, we should also be able to use them where possible on the read path.

Some jobs have a capped local disk, and so being able to quickly fetch these back rather than rebuild them is very useful for making lookup joins lower latency. For now, we'll keep things simple - this should only run when lookup files are being persisted to object store, deletion vectors are off (i.e. MOR tables), and we can actually use the SSTable files persisted by compaction jobs (the join consumer wants the full row schema as opposed to a projection of the data).

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the read path for lookup files and the code that persists index files to object storage. Confirm how MOR tables, deletion vectors, compaction-produced SSTable files, and full-row-schema lookup joins are represented; done means eligible persisted SSTables are fetched instead of rebuilt under the stated conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.