[Feature] Download remote index files when already built
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Now that Paimon enables remote index files for data, we should also be able to use them where possible on the read path.
Some jobs have a capped local disk, and so being able to quickly fetch these back rather than rebuild them is very useful for making lookup joins lower latency. For now, we'll keep things simple - this should only run when lookup files are being persisted to object store, deletion vectors are off (i.e. MOR tables), and we can actually use the SSTable files persisted by compaction jobs (the join consumer wants the full row schema as opposed to a projection of the data).
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the read path for lookup files and the code that persists index files to object storage. Confirm how MOR tables, deletion vectors, compaction-produced SSTable files, and full-row-schema lookup joins are represented; done means eligible persisted SSTables are fetched instead of rebuilt under the stated conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100