Partitioning table by `_hoodie_partition_path` fails
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, creating a table partitioned by "_hoodie_partition_path" using Glue catalog fails w/ the following exception:
{code:java}
AnalysisException: Found duplicate column(s) in the data schema and the partition schema: _hoodie_partition_path
{code}
Using following DDL:
{code:java}
CREATE EXTERNAL TABLE `active_storage_attachments`( `_hoodie_commit_time` string COMMENT '', `_hoodie_commit_seqno` string COMMENT '', `_hoodie_record_key` string COMMENT '', `_hoodie_file_name` string COMMENT '', `_change_operation_type` string COMMENT '', `_upstream_event_processed_ts_ms` bigint COMMENT '', `db_shard_source_partition` string COMMENT '', `_event_origin_ts_ms` bigint COMMENT '', `_event_tx_id` bigint COMMENT '', `_event_lsn` bigint COMMENT '', `_event_xmin` bigint COMMENT '', `id` bigint COMMENT '', `name` string COMMENT '', `record_type` string COMMENT '', `record_id` bigint COMMENT '', `blob_id` bigint COMMENT '', `created_at` timestamp COMMENT '')PARTITIONED BY ( `_hoodie_partition_path` string COMMENT '')ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' WITH SERDEPROPERTIES ( 'hoodie.query.as.ro.table'='false', 'path'='...') STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'LOCATION '...'
TBLPROPERTIES ( 'spark.sql.sources.provider'='hudi' )
{code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4626
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1297
- Affects version(s):
- 0.12.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the supplied CREATE EXTERNAL TABLE DDL with the Glue catalog and the `_hoodie_partition_path` partition column. Trace the schema handling that produces the duplicate-column AnalysisException, then verify that the table can be created without the data and partition schemas conflicting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100