apache / apache/iceberg

Spark: Support ALTER TABLE ... DROP PARTITION syntax

Open
#16,670 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

# Motivation
Iceberg has no SQL-level way to drop an existing partition from a table. `ALTER TABLE … DROP PARTITION` is a standard Hive/Spark/Trino DDL statement that users coming from those ecosystems expect to work, and not having it forces users to translate partition values back into predicates on source columns.

## How users drop a partition today
The only SQL option is a row-targeting DELETE keyed on the source column:
```sql
-- "drop" January 2024 from a table partitioned by months(ts)
DELETE FROM events
WHERE ts >= TIMESTAMP '2024-01-01' AND ts < TIMESTAMP '2024-02-01';
```
Problems:
- The table is partitioned by months(ts) (partition field ts_month), but the user has no SQL way to address that partition directly. So, they have to translate "January 2024" back into a half-open range on the source column.

# Proposed feature
Add native support for the standard ALTER TABLE … DROP PARTITION syntax, addressing the partition field name (not the source column), in the same human-readable form Iceberg already writes to manifest partition paths:
```sql
ALTER TABLE events DROP PARTITION (ts_year = '2024');
ALTER TABLE events DROP PARTITION (ts_month = '2024-01');
ALTER TABLE events DROP PARTITION (ts_day = '2024-02-15');
ALTER TABLE events DROP PARTITION (ts_hour = '2024-01-15-11');
ALTER TABLE events DROP PARTITION (id_bucket = 0);
ALTER TABLE events DROP PARTITION (region = 'eu', dt = '2024-01-01');
ALTER TABLE events DROP PARTITION IF EXISTS (data = 'missing');

-- Atomic multi-partition drop (SupportsAtomicPartitionManagement)
ALTER TABLE events DROP PARTITION (data = 'b'), PARTITION (data = 'd');
```

# Approach comparison

| Concern | DELETE FROM … WHERE … (today) | ALTER TABLE … DROP PARTITION (proposed) |
|----------|------------------------------|------------------------------------------|
| SQL surface | Predicates | Standard Spark DDL |
| Naming the partition | Source-column range arithmetic | Direct partition-field name (`ts_month`, `id_bucket`, …) |
| Bucket / truncate partitions | Cannot express cleanly in SQL | `id_bucket = 0` names the partition directly |
| Time-transform partitions | User computes a `[lo, hi)` range each time | Same `"yyyy-MM"` / `"yyyy-MM-dd"` / `"yyyy-MM-dd-HH"` form Iceberg writes to manifests |
| Migration from Spark tables | Pipelines using `DROP PARTITION` need rewriting | Drop-in source compatible |

# Implementation sketch
- SparkTable now implements `SupportsAtomicPartitionManagement`. `partitionSchema(...)` exposes the partition fields (ts_day, id_bucket, …) with their human-readable types: Atring for year/month/day/hour, integer for bucket, source type for identity/truncate.
- `dropPartition(InternalRow)` builds a transform-aware predicate (e.g. `Expressions.equal(Expressions.day("ts"), Transforms.parseHumanDay("2024-02-15"))`) and commits it via `DeleteFiles.deleteFromRowFilter(...)` a manifest-level operation.
- new `Transforms.parseHumanYear/Month/Day/Hour(String)` → int static methods on org.apache.iceberg.transforms.Transforms, paired with the existing `TransformUtil.humanYear/Month/Day/Hour` forward conversions. Centralizes the human-string ↔ int-ordinal logic.

# Safety
`deleteFromRowFilter(...)` already runs the strict-projection check at commit, so partial-file deletes are rejected by Iceberg core; the new API surface cannot silently remove rows outside the targeted partition.

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.