apache / apache/spark

[PIPELINES] Support partition transforms in SDP partition_cols

Open
#57,908 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

spark=4.1.3
iceberg=1.11.0

### Problem

SDP currently treats every entry in `partition_cols` as an identity partition transform.

for example

```python
@dp.materialized_view(partition_cols=["months(event_time)"])
def events():
...
```

causes `months(event_time)` to be interpreted as a column name instead of a partition transform.

Behaviour comes from `DatasetManager.materializeTable` where all partition cols are converted using identity

```scala
table.partitionCols.toSeq.flatten.map(Expressions.identity)
```

and as a result using the above partition col it just becomes `identity('months(event_time)')`.

### Expected behaviour

SDP should support the same partition transforms that Spark SQL already supports

Bare col-names (`partition_col=["event_date"]`) should continue producing `identity(event_date)`
but partition transforms should be respected, and combining partitions like

```python
partition_cols=["country", "bucket(16, user_id)"]
```

should produce its corresponding V2 partition transforms.

### Existing SQL support

Catalyst parses partition transform syntax for SQL `PARTITIONED BY` and creates corresponding V2 `Transform`

SDP should just reuse the existing Catalyst parsing logic and not short-circuit everything to identity transform.

Contributor guide

Open the contributing guide

Research direction

Start in DatasetManager.materializeTable, where partition_cols are currently mapped through Expressions.identity. Read the existing Catalyst parsing logic used for SQL PARTITIONED BY transforms, then verify the behavior for bare columns, months(event_time), and bucket(16, user_id). Done means identity transforms remain for column names while supported transform syntax produces the corresponding V2 transforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.