Null partition handling of hive migration
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.0
### Query engine
Spark3.5
### Please describe the bug 🐞
I tried to migrate a TPCDS hive table to iceberg using
`CALL spark_catalog.system.migrate('tpcdsdbiceberg_10tb_partitioned2_perfteam.catalog_sales`. The null partition is stored as `cs_sold_date_sk=__HIVE_DEFAULT_PARTITION__` on S3.
After migration, I get below strange result:
```sql
select count(*) from catalog_sales
14399880363
select count(*) from catalog_sales_BACKUP_
14399880363
```
```sql
select count(*) from catalog_sales where cs_sold_date_sk is null
0
select count(*) from catalog_sales_BACKUP_ where cs_sold_date_sk is null
71981198
```
```sql
select count(*) from catalog_sales where cs_sold_date_sk is not null
14327899165
select count(*) from catalog_sales_BACKUP_ where cs_sold_date_sk is not null
14327899165
```
Is it a known issue? I didn't search anything.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with the Spark 3.5 migration entry point shown in the CALL spark_catalog.system.migrate(...) example and reproduce the case using the TPCDS Hive table with a null partition stored as __HIVE_DEFAULT_PARTITION__ on S3. Compare the migrated table with catalog_sales_BACKUP_ using the three count queries from the report. Done means null-partitioned rows remain queryable as NULL after migration while non-null rows and total counts are preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100