apache / apache/iceberg

MR: IcebergInputFormat drops identity partition constants

Closed Beginner friendly
#18,033 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

**Apache Iceberg version**
main @ 9d7b2c52f

**Query engine**
None — the bug is in the MapReduce/Hive input format (`iceberg-mr`).

**Please describe the bug**
`IcebergInputFormat.IcebergRecordReader.openTask()` (`mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java` line 310) builds the reader without `idToConstant(...)`, so identity partition values recorded only in the manifest are never handed to the reader. Reading such a file returns `null` in the partition columns.

This is a regression from a2802c44c (#15333), which merged the Avro, ORC and Parquet branches into one `ReadBuilder`. Before that change all three passed `constantsMap(task, IdentityPartitionConverters::convertConstant)`. The sibling reader touched by the same PR, `data/src/main/java/org/apache/iceberg/data/GenericReader.openFile()`, kept `.idToConstant(partition)`.

**Steps to reproduce**
1. Create an identity-partitioned table.
2. Add a data file that does not physically contain the partition columns, with the partition values set on the manifest entry. `TableMigrationUtil.listPartition(...)` produces exactly this — it takes partition values from Hive metadata, not from the file — so any table imported with `add_files` or `migrate` is in this state.
3. Read the table through `IcebergInputFormat`.

Expected: the partition columns hold the manifest values. Actual: they are `null`.

**Additional context**
N/A

Contributor guide

Open the contributing guide

Research direction

Start in mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java at IcebergRecordReader.openTask(), then compare it with data/src/main/java/org/apache/iceberg/data/GenericReader.openFile(). Reproduce the identity-partitioned case described using manifest-only partition values and verify that IcebergInputFormat returns those values in the partition columns instead of null.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.