apache / apache/iceberg

Null partition handling of hive migration

Open
#15,332 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.0

### Query engine

Spark3.5

### Please describe the bug 🐞

I tried to migrate a TPCDS hive table to iceberg using
`CALL spark_catalog.system.migrate('tpcdsdbiceberg_10tb_partitioned2_perfteam.catalog_sales`. The null partition is stored as `cs_sold_date_sk=__HIVE_DEFAULT_PARTITION__` on S3.

After migration, I get below strange result:

```sql
select count(*) from catalog_sales
14399880363
select count(*) from catalog_sales_BACKUP_
14399880363
```

```sql
select count(*) from catalog_sales where cs_sold_date_sk is null
0
select count(*) from catalog_sales_BACKUP_ where cs_sold_date_sk is null
71981198
```

```sql
select count(*) from catalog_sales where cs_sold_date_sk is not null
14327899165
select count(*) from catalog_sales_BACKUP_ where cs_sold_date_sk is not null
14327899165
```

Is it a known issue? I didn't search anything.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with the Spark 3.5 migration entry point shown in the CALL spark_catalog.system.migrate(...) example and reproduce the case using the TPCDS Hive table with a null partition stored as __HIVE_DEFAULT_PARTITION__ on S3. Compare the migrated table with catalog_sales_BACKUP_ using the three count queries from the report. Done means null-partitioned rows remain queryable as NULL after migration while non-null rows and total counts are preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.