apache / apache/iceberg

Metadata: Hidden Partitions with transformation depending on the way it has created has different names

Open
#16,238 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.0

### Query engine

Spark

### Please describe the bug 🐞

Hi,

We have in my project a way to detect if there is new partition column in the ETL process that writes into iceberg that if we want to add/drop a column into the partitions we `ALTER TABLE ADD/DROP PARTITION FIELD`. However, when we have a hidden partition with a bucket transformation if it is created for the first time via Pyspark createOrReplace in the metadata json it has the name "**column_name_bucket**".

However, if the partition didn't exist before and we do an `ALTER TABLE` and added the partitions it is now named "**column_name_bucket_n**", being n the n from **bucket(column_name, n)**.

This also happens when doing in athena queries:
`SELECT * FROM "database"."table$partitions" limit 10;`

This is causing issues how to properly identify if the partition was added since we have 2 different ways to detect. Using both seems risky in my perspective

If possible column_name_bucket_n nomenclature is perfect for us and if it can be standardized. I also don't know if it is an issue here or in the Spark on how it handles the write if it is on the Spark side I can open the issue there

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Reproduce the hidden bucket-partition metadata difference between PySpark createOrReplace and ALTER TABLE ADD PARTITION FIELD, then inspect the partition metadata exposed through the Athena $partitions query. Trace where each path assigns the partition name and determine whether Iceberg or Spark owns the naming; done means both creation paths use one consistent nomenclature.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.