apache / apache/iceberg

The estimated table size is inaccurate

Open
#15,684 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.1 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

Hi, team!
The SparkSchemaUtil. estimateSize method calculates the size based on the default size of the field type and the number of rows, may differ significantly from the actual size.
May I ask if there are any areas that can be improved?
https://github.com/apache/iceberg/blob/eb460a524f3a192e3584a3bdabcff237c38b04a4/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java#L339

The discovery of this issue is due to the different execution plans of Spark querying the parquet source/iceberg tables.
Spark default field size percentage based on file size:
https://github.com/apache/spark/blob/10dd228d4c09166c2cb744cb0e3e7f15385afae0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala#L34

May I ask if it is possible to use the manifest file for more accurate statistics?

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with SparkSchemaUtil.java at the estimateSize method and compare its calculation with Spark's SizeInBytesOnlyStatsPlanVisitor reference. Investigate whether Iceberg manifest information can provide more accurate table-size statistics for Spark query planning. Done means the estimated size better reflects the actual table size and avoids the reported execution-plan discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.