MR: IcebergInputFormat drops identity partition constants
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
**Apache Iceberg version**
main @ 9d7b2c52f
**Query engine**
None — the bug is in the MapReduce/Hive input format (`iceberg-mr`).
**Please describe the bug**
`IcebergInputFormat.IcebergRecordReader.openTask()` (`mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java` line 310) builds the reader without `idToConstant(...)`, so identity partition values recorded only in the manifest are never handed to the reader. Reading such a file returns `null` in the partition columns.
This is a regression from a2802c44c (#15333), which merged the Avro, ORC and Parquet branches into one `ReadBuilder`. Before that change all three passed `constantsMap(task, IdentityPartitionConverters::convertConstant)`. The sibling reader touched by the same PR, `data/src/main/java/org/apache/iceberg/data/GenericReader.openFile()`, kept `.idToConstant(partition)`.
**Steps to reproduce**
1. Create an identity-partitioned table.
2. Add a data file that does not physically contain the partition columns, with the partition values set on the manifest entry. `TableMigrationUtil.listPartition(...)` produces exactly this — it takes partition values from Hive metadata, not from the file — so any table imported with `add_files` or `migrate` is in this state.
3. Read the table through `IcebergInputFormat`.
Expected: the partition columns hold the manifest values. Actual: they are `null`.
**Additional context**
N/A
Contributor guide
Research direction
Start in mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java at IcebergRecordReader.openTask(), then compare it with data/src/main/java/org/apache/iceberg/data/GenericReader.openFile(). Reproduce the identity-partitioned case described using manifest-only partition values and verify that IcebergInputFormat returns those values in the partition columns instead of null.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100