apache / apache/iceberg

Skip empty files while migrating tables to iceberg

Open
#17,926 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

ORC:

```
java.lang.IllegalArgumentException: ORC schema does not contain Iceberg IDs
at org.apache.iceberg.orc.ORCSchemaUtil.convert(ORCSchemaUtil.java:293)
at org.apache.iceberg.orc.OrcMetrics.buildOrcMetrics(OrcMetrics.java:143)
at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:93)
at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:87)
at org.apache.iceberg.data.TableMigrationUtil.getOrcMetrics(TableMigrationUtil.java:296)
at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$4(TableMigrationUtil.java:244)
```

Parquet:

```
java.lang.RuntimeException: mig_parquet_sample/date_key=2026-09-02/1_$folder$ is not a Parquet file (length is too low: 0)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:599)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.(ParquetFileReader.java:971)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.(ParquetFileReader.java:961)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:718)
at org.apache.iceberg.parquet.ParquetUtil.fileMetrics(ParquetUtil.java:64)
at org.apache.iceberg.data.TableMigrationUtil.getParquetMetrics(TableMigrationUtil.java:287)
at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$3(TableMigrationUtil.java:236)

```

Sometimes empty files are created in hive, we need to manually remove all empty files before migrating to iceberg.

We can fix this by skipping empty files

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Read TableMigrationUtil, especially listPartition and its getOrcMetrics and getParquetMetrics paths shown in the stack traces. Trace how migration handles discovered files, then verify that zero-length files are skipped before ORC or Parquet metrics are read; done means migrations no longer fail on empty files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.