apache / apache/hudi

Partitioning table by `_hoodie_partition_path` fails

Open
#15,340 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, creating a table partitioned by "_hoodie_partition_path" using Glue catalog fails w/ the following exception:
{code:java}
AnalysisException: Found duplicate column(s) in the data schema and the partition schema: _hoodie_partition_path
{code}
Using following DDL:
{code:java}
CREATE EXTERNAL TABLE `active_storage_attachments`(  `_hoodie_commit_time` string COMMENT '',   `_hoodie_commit_seqno` string COMMENT '',   `_hoodie_record_key` string COMMENT '',   `_hoodie_file_name` string COMMENT '',   `_change_operation_type` string COMMENT '',   `_upstream_event_processed_ts_ms` bigint COMMENT '',   `db_shard_source_partition` string COMMENT '',   `_event_origin_ts_ms` bigint COMMENT '',   `_event_tx_id` bigint COMMENT '',   `_event_lsn` bigint COMMENT '',   `_event_xmin` bigint COMMENT '',   `id` bigint COMMENT '',   `name` string COMMENT '',   `record_type` string COMMENT '',   `record_id` bigint COMMENT '',   `blob_id` bigint COMMENT '',   `created_at` timestamp COMMENT '')PARTITIONED BY (   `_hoodie_partition_path` string COMMENT '')ROW FORMAT SERDE   'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' WITH SERDEPROPERTIES (   'hoodie.query.as.ro.table'='false',   'path'='...') STORED AS INPUTFORMAT   'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT   'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'LOCATION  '...'

TBLPROPERTIES (  'spark.sql.sources.provider'='hudi' )
{code}
 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4626
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1297
- Affects version(s):
- 0.12.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the supplied CREATE EXTERNAL TABLE DDL with the Glue catalog and the `_hoodie_partition_path` partition column. Trace the schema handling that produces the duplicate-column AnalysisException, then verify that the table can be created without the data and partition schemas conflicting.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.