apache / apache/hudi

Support common Spark transformations w/in Spark SQL "partitioned by" clause

Open
#15,232 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently if you create a Hudi table from Spark SQL:
{code:java}
CREATE TABLE test_create(
f1 STRING,
f2 STRING,
f3 STRING,
ts timestamp
)using hudi
partitioned by ( hours(ts))
options (
type = 'mor'
)
tblproperties (
primaryKey = 'f1',
preCombineField = 'ts'
);
{code}
You'll be getting
{code:java}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.spark.sql.AnalysisException: Transforms cannot be converted to partition columns: hours(ts){code}
 

Original reported task:

[https://github.com/apache/hudi/issues/5810]

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4244
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1658

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark SQL table-creation and partition-column handling paths, then review the linked JIRA issue HUDI-4244 and the original task. Reproduce the example using partitioned by (hours(ts)) and identify the existing tests around partition transforms. Done means common Spark transformations are accepted without the reported AnalysisException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.