apache / apache/gravitino

[Improvement] leverage concurrent iteration of fileScanTasks to improve table scan plan performance in Iceberg REST server

Open
#9,353 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

for (FileScanTask fileScanTask : fileScanTasks) —— would using a parallel stream here be better? During actual usage we found this block to be unstable in terms of latency, and our trace revealed the costs is below
+---[min=0.002104ms,max=3137.10955ms,total=3159.839086ms,count=476] java.util.Iterator:hasNext()
#9047

### How should we improve?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the for-loop over FileScanTasks in the Iceberg REST server's table scan plan path and review issue #9047 for context. Compare sequential and concurrent iteration using the reported latency trace or a suitable benchmark; done means improved and more stable scan-plan latency without changing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.