Metadata: Hidden Partitions with transformation depending on the way it has created has different names
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.0
### Query engine
Spark
### Please describe the bug 🐞
Hi,
We have in my project a way to detect if there is new partition column in the ETL process that writes into iceberg that if we want to add/drop a column into the partitions we `ALTER TABLE ADD/DROP PARTITION FIELD`. However, when we have a hidden partition with a bucket transformation if it is created for the first time via Pyspark createOrReplace in the metadata json it has the name "**column_name_bucket**".
However, if the partition didn't exist before and we do an `ALTER TABLE` and added the partitions it is now named "**column_name_bucket_n**", being n the n from **bucket(column_name, n)**.
This also happens when doing in athena queries:
`SELECT * FROM "database"."table$partitions" limit 10;`
This is causing issues how to properly identify if the partition was added since we have 2 different ways to detect. Using both seems risky in my perspective
If possible column_name_bucket_n nomenclature is perfect for us and if it can be standardized. I also don't know if it is an issue here or in the Spark on how it handles the write if it is on the Spark side I can open the issue there
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Reproduce the hidden bucket-partition metadata difference between PySpark createOrReplace and ALTER TABLE ADD PARTITION FIELD, then inspect the partition metadata exposed through the Athena $partitions query. Trace where each path assigns the partition name and determine whether Iceberg or Spark owns the naming; done means both creation paths use one consistent nomenclature.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100