apache / apache/gluten

Avoid retrieving the partition schema of Iceberg repeatedly.

Open
#7,376 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

When there are a large number of partitions and files, we repeatedly retrieve the partition schema each time, which is very costly. Internal testing shows that when the number of partitions exceeds ten thousand, this process can take anywhere from several minutes to hours.
![image](https://github.com/user-attachments/assets/bb7fe6ae-f97a-4b5e-b235-98e0db69be5c)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.