Support common Spark transformations w/in Spark SQL "partitioned by" clause
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently if you create a Hudi table from Spark SQL:
{code:java}
CREATE TABLE test_create(
f1 STRING,
f2 STRING,
f3 STRING,
ts timestamp
)using hudi
partitioned by ( hours(ts))
options (
type = 'mor'
)
tblproperties (
primaryKey = 'f1',
preCombineField = 'ts'
);
{code}
You'll be getting
{code:java}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.spark.sql.AnalysisException: Transforms cannot be converted to partition columns: hours(ts){code}
Original reported task:
[https://github.com/apache/hudi/issues/5810]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4244
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1658
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Spark SQL table-creation and partition-column handling paths, then review the linked JIRA issue HUDI-4244 and the original task. Reproduce the example using partitioned by (hours(ts)) and identify the existing tests around partition transforms. Done means common Spark transformations are accepted without the reported AnalysisException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100