apache / apache/iceberg

Make Spark readers function asynchronously for many small files.

Open
#15,287 6 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

In Spark readers, scan tasks are currently processed sequentially. The iteration logic in `BaseReader.next()` opens one task at a time, fully consumes it, and only then proceeds to the next task.

https://github.com/apache/iceberg/blob/f49b2fd97b48682d4e4ca6f1a552cb48f53c4ea5/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java#L131-L145

With a large number of small files let's say hundreds or thousands of 5–10 KB files, this sequential task processing can lead to significant overhead.
Each task is opened and read independently, which may underutilize available CPU and I/O parallelism, especially on object stores with non-trivial per request latency.

Possible Improvement:
It may be beneficial to optionally allow Spark readers to process multiple small-file tasks concurrently, buffering rows into a shared iterator for downstream processing, while preserving the existing sequential behavior by default.

### Query engine

None

### Willingness to contribute

- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community

Contributor guide

Open the contributing guide

Research direction

Start with spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java, especially the iteration logic at lines 131-145. Establish how an optional concurrent path could preserve sequential behavior by default, and define the tests or benchmarks needed to verify buffered processing of many small files and the existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.