apache / apache/hudi

[SUPPORT] `CREATE TABLE ... USING hudi` DDL does not preserve partitioning order when syncing to AWS Glue

Open
#10,182 5 comments 1 reaction 0 assignees View on GitHub
area:aws component:catalog-sync priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

We've observed that the `CREATE TABLE` DDL alphabetizes partition column names when syncing to Glue. The values in `hoodie.properties` are correct; this seems to only affect the Glue table. While this doesn't impact reads from Spark, it seems that it causes issues for Trino.

**To Reproduce**

Steps to reproduce the behavior:

1. Create a Hudi table with the following code. Note that the partitioning columns are specified in `c, a, b` order.
```python
df = spark.createDataFrame([{"a": 1, "b": 1, "c": 1, "d": 1}, {"a": 2, "b": 2, "c": 2, "d": 1}])

location = "s3://..."

df.write.format("hudi").options(
**{
'hoodie.bootstrap.index.enable': 'false',
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.CustomKeyGenerator',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.partitionpath.field': 'c:SIMPLE,a:SIMPLE,b:SIMPLE',
'hoodie.datasource.write.precombine.field': 'd',
'hoodie.datasource.write.recordkey.field': 'd',
'hoodie.datasource.write.table.name': 'test_nonalpha_partitioning',
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.table.name': 'test_nonalpha_partitioning',
}
).save(location)

spark.sql(f"""
create table prototype_lakehouse_testing.test_nonalpha_partitioning
using hudi
location '{location}'
""")
```
2. Observe that the Glue table reports partition columns in alphabetical order:
```
❯ aws glue get-table --database-name 'prototype_lakehouse_testing' --name 'test_nonalpha_partitioning' | jq '.Table.PartitionKeys'
[
{
"Name": "a",
"Type": "bigint"
},
{
"Name": "b",
"Type": "bigint"
},
{
"Name": "c",
"Type": "bigint"
}
]
```
while the table's `hoodie.properties` reports `hoodie.table.partition.fields=c,a,b`

**Expected behavior**

We expect the Glue table to preserve the partition column order.

**Environment Description**

The above was run on an AWS EMR cluster running version `emr-6.10.1`

* Hudi version : `0.12.2-amzn-0`

* Spark version : `3.3.1`

* Hive version `3.1.3`

* Hadoop version : `3.3.3`

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : Spark on Docker

**Additional context**

Add any other context about the problem here.

**Stacktrace**

n/a

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the CREATE TABLE ... USING hudi example and compare hoodie.properties with the AWS Glue PartitionKeys returned by aws glue get-table. Trace the Glue synchronization entry point for this DDL and verify that the partition order is preserved as c,a,b when the issue is fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, java, python, spark
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.