Make Spark readers function asynchronously for many small files.
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
In Spark readers, scan tasks are currently processed sequentially. The iteration logic in `BaseReader.next()` opens one task at a time, fully consumes it, and only then proceeds to the next task.
https://github.com/apache/iceberg/blob/f49b2fd97b48682d4e4ca6f1a552cb48f53c4ea5/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java#L131-L145
With a large number of small files let's say hundreds or thousands of 5–10 KB files, this sequential task processing can lead to significant overhead.
Each task is opened and read independently, which may underutilize available CPU and I/O parallelism, especially on object stores with non-trivial per request latency.
Possible Improvement:
It may be beneficial to optionally allow Spark readers to process multiple small-file tasks concurrently, buffering rows into a shared iterator for downstream processing, while preserving the existing sequential behavior by default.
### Query engine
None
### Willingness to contribute
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
Contributor guide
Research direction
Start with spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java, especially the iteration logic at lines 131-145. Establish how an optional concurrent path could preserve sequential behavior by default, and define the tests or benchmarks needed to verify buffered processing of many small files and the existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100