[PIPELINES] Support partition transforms in SDP partition_cols
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
spark=4.1.3
iceberg=1.11.0
### Problem
SDP currently treats every entry in `partition_cols` as an identity partition transform.
for example
```python
@dp.materialized_view(partition_cols=["months(event_time)"])
def events():
...
```
causes `months(event_time)` to be interpreted as a column name instead of a partition transform.
Behaviour comes from `DatasetManager.materializeTable` where all partition cols are converted using identity
```scala
table.partitionCols.toSeq.flatten.map(Expressions.identity)
```
and as a result using the above partition col it just becomes `identity('months(event_time)')`.
### Expected behaviour
SDP should support the same partition transforms that Spark SQL already supports
Bare col-names (`partition_col=["event_date"]`) should continue producing `identity(event_date)`
but partition transforms should be respected, and combining partitions like
```python
partition_cols=["country", "bucket(16, user_id)"]
```
should produce its corresponding V2 partition transforms.
### Existing SQL support
Catalyst parses partition transform syntax for SQL `PARTITIONED BY` and creates corresponding V2 `Transform`
SDP should just reuse the existing Catalyst parsing logic and not short-circuit everything to identity transform.
Contributor guide
Research direction
Start in DatasetManager.materializeTable, where partition_cols are currently mapped through Expressions.identity. Read the existing Catalyst parsing logic used for SQL PARTITIONED BY transforms, then verify the behavior for bare columns, months(event_time), and bucket(16, user_id). Done means identity transforms remain for column names while supported transform syntax produces the corresponding V2 transforms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100