Skip empty files while migrating tables to iceberg
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
ORC:
```
java.lang.IllegalArgumentException: ORC schema does not contain Iceberg IDs
at org.apache.iceberg.orc.ORCSchemaUtil.convert(ORCSchemaUtil.java:293)
at org.apache.iceberg.orc.OrcMetrics.buildOrcMetrics(OrcMetrics.java:143)
at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:93)
at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:87)
at org.apache.iceberg.data.TableMigrationUtil.getOrcMetrics(TableMigrationUtil.java:296)
at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$4(TableMigrationUtil.java:244)
```
Parquet:
```
java.lang.RuntimeException: mig_parquet_sample/date_key=2026-09-02/1_$folder$ is not a Parquet file (length is too low: 0)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:599)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.(ParquetFileReader.java:971)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.(ParquetFileReader.java:961)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:718)
at org.apache.iceberg.parquet.ParquetUtil.fileMetrics(ParquetUtil.java:64)
at org.apache.iceberg.data.TableMigrationUtil.getParquetMetrics(TableMigrationUtil.java:287)
at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$3(TableMigrationUtil.java:236)
```
Sometimes empty files are created in hive, we need to manually remove all empty files before migrating to iceberg.
We can fix this by skipping empty files
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Read TableMigrationUtil, especially listPartition and its getOrcMetrics and getParquetMetrics paths shown in the stack traces. Trace how migration handles discovered files, then verify that zero-length files are skipped before ORC or Parquet metrics are read; done means migrations no longer fail on empty files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100