The estimated table size is inaccurate
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Hi, team!
The SparkSchemaUtil. estimateSize method calculates the size based on the default size of the field type and the number of rows, may differ significantly from the actual size.
May I ask if there are any areas that can be improved?
https://github.com/apache/iceberg/blob/eb460a524f3a192e3584a3bdabcff237c38b04a4/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java#L339
The discovery of this issue is due to the different execution plans of Spark querying the parquet source/iceberg tables.
Spark default field size percentage based on file size:
https://github.com/apache/spark/blob/10dd228d4c09166c2cb744cb0e3e7f15385afae0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala#L34
May I ask if it is possible to use the manifest file for more accurate statistics?
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with SparkSchemaUtil.java at the estimateSize method and compare its calculation with Spark's SizeInBytesOnlyStatsPlanVisitor reference. Investigate whether Iceberg manifest information can provide more accurate table-size statistics for Spark query planning. Done means the estimated size better reflects the actual table size and avoids the reported execution-plan discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100