Spark: Support ALTER TABLE ... DROP PARTITION syntax
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
# Motivation
Iceberg has no SQL-level way to drop an existing partition from a table. `ALTER TABLE … DROP PARTITION` is a standard Hive/Spark/Trino DDL statement that users coming from those ecosystems expect to work, and not having it forces users to translate partition values back into predicates on source columns.
## How users drop a partition today
The only SQL option is a row-targeting DELETE keyed on the source column:
```sql
-- "drop" January 2024 from a table partitioned by months(ts)
DELETE FROM events
WHERE ts >= TIMESTAMP '2024-01-01' AND ts < TIMESTAMP '2024-02-01';
```
Problems:
- The table is partitioned by months(ts) (partition field ts_month), but the user has no SQL way to address that partition directly. So, they have to translate "January 2024" back into a half-open range on the source column.
# Proposed feature
Add native support for the standard ALTER TABLE … DROP PARTITION syntax, addressing the partition field name (not the source column), in the same human-readable form Iceberg already writes to manifest partition paths:
```sql
ALTER TABLE events DROP PARTITION (ts_year = '2024');
ALTER TABLE events DROP PARTITION (ts_month = '2024-01');
ALTER TABLE events DROP PARTITION (ts_day = '2024-02-15');
ALTER TABLE events DROP PARTITION (ts_hour = '2024-01-15-11');
ALTER TABLE events DROP PARTITION (id_bucket = 0);
ALTER TABLE events DROP PARTITION (region = 'eu', dt = '2024-01-01');
ALTER TABLE events DROP PARTITION IF EXISTS (data = 'missing');
-- Atomic multi-partition drop (SupportsAtomicPartitionManagement)
ALTER TABLE events DROP PARTITION (data = 'b'), PARTITION (data = 'd');
```
# Approach comparison
| Concern | DELETE FROM … WHERE … (today) | ALTER TABLE … DROP PARTITION (proposed) |
|----------|------------------------------|------------------------------------------|
| SQL surface | Predicates | Standard Spark DDL |
| Naming the partition | Source-column range arithmetic | Direct partition-field name (`ts_month`, `id_bucket`, …) |
| Bucket / truncate partitions | Cannot express cleanly in SQL | `id_bucket = 0` names the partition directly |
| Time-transform partitions | User computes a `[lo, hi)` range each time | Same `"yyyy-MM"` / `"yyyy-MM-dd"` / `"yyyy-MM-dd-HH"` form Iceberg writes to manifests |
| Migration from Spark tables | Pipelines using `DROP PARTITION` need rewriting | Drop-in source compatible |
# Implementation sketch
- SparkTable now implements `SupportsAtomicPartitionManagement`. `partitionSchema(...)` exposes the partition fields (ts_day, id_bucket, …) with their human-readable types: Atring for year/month/day/hour, integer for bucket, source type for identity/truncate.
- `dropPartition(InternalRow)` builds a transform-aware predicate (e.g. `Expressions.equal(Expressions.day("ts"), Transforms.parseHumanDay("2024-02-15"))`) and commits it via `DeleteFiles.deleteFromRowFilter(...)` a manifest-level operation.
- new `Transforms.parseHumanYear/Month/Day/Hour(String)` → int static methods on org.apache.iceberg.transforms.Transforms, paired with the existing `TransformUtil.humanYear/Month/Day/Hour` forward conversions. Centralizes the human-string ↔ int-ordinal logic.
# Safety
`deleteFromRowFilter(...)` already runs the strict-projection check at commit, so partial-file deletes are rejected by Iceberg core; the new API surface cannot silently remove rows outside the targeted partition.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Assessment
This issue has not been assessed yet.