apache / apache/hudi

Support YYYY/MM/DD partition format with hive

Open
#17,282 2 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently a format like YYYY/MM/DD fails when syncing with hive. The Jira aims to add a fix so that such a format is supported.
Steps to reproduce: The table created below uses a custom keygen with combination of simple and timestamp keygen. Timestamp keygen produces an output of format - YYYY/MM/DD
{code:java}
import org.apache.hudi.HoodieSparkUtils
import org.apache.hudi.common.config.TypedProperties
import org.apache.hudi.common.util.StringUtils
import org.apache.hudi.exception.HoodieException
import org.apache.hudi.functional.TestSparkSqlWithCustomKeyGenerator._
import org.apache.hudi.testutils.HoodieClientTestUtils.createMetaClient
import org.apache.hudi.util.SparkKeyGenUtilsimport org.apache.spark.sql.SaveMode
import org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase
import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat
import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertTrue}
import org.slf4j.LoggerFactory
    val df = spark.sql(
      s"""SELECT 1 as id, 'a1' as name, 1.6 as price, 1704121827 as ts, 'cat1' as segment
         | UNION
         | SELECT 2 as id, 'a2' as name, 10.8 as price, 1704121827 as ts, 'cat1' as segment
         | UNION
         | SELECT 3 as id, 'a3' as name, 30.0 as price, 1706800227 as ts, 'cat1' as segment
         | UNION
         | SELECT 4 as id, 'a4' as name, 103.4 as price, 1701443427 as ts, 'cat2' as segment
         | UNION
         | SELECT 5 as id, 'a5' as name, 1999.0 as price, 1704121827 as ts, 'cat2' as segment
         | UNION
         | SELECT 6 as id, 'a6' as name, 80.0 as price, 1704121827 as ts, 'cat3' as segment
         |""".stripMargin)    df.write.format("hudi").option("hoodie.datasource.write.table.type", "MERGE_ON_READ").option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.CustomAvroKeyGenerator").option("hoodie.datasource.write.partitionpath.field", "segment:simple,ts:timestamp").option("hoodie.datasource.write.recordkey.field", "id").option("hoodie.datasource.write.precombine.field", "name").option("hoodie.table.name", "hudi_table_2").option("hoodie.insert.shuffle.parallelism", "1").option("hoodie.upsert.shuffle.parallelism", "1").option("hoodie.bulkinsert.shuffle.parallelism", "1").option("hoodie.keygen.timebased.timestamp.type", "SCALAR").option("hoodie.keygen.timebased.output.dateformat", "yyyy/MM/DD").option("hoodie.keygen.timebased.timestamp.scalar.time.unit", "seconds").mode(SaveMode.Overwrite).save("/user/hive/warehouse/hudi_table_2")

// Sync with hive
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
  --jdbc-url jdbc:hive2://hiveserver:10000 \
  --user hive \
  --pass hive \
  --partitioned-by segment,ts \
  --base-path /user/hive/warehouse/hudi_table_2 \
  --database default \
  --table hudi_table_2 \
  --partition-value-extractor org.apache.hudi.hive.MultiPartKeysValueExtractor   {code}
 

Hive creation fails now.
{code:java}
2024-10-06 14:33:44,200 INFO  [main] hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_table_2
    at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:180)
    at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:547)
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: failed to sync the table hudi_table_2_ro
    at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:272)
    at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:203)
    at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:177)
    ... 1 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_table_2_ro
    at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:474)
    at org.apache.hudi.hive.HiveSyncTool.validateAndSyncPartitions(HiveSyncTool.java:321)
    at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:261)
    ... 3 more
Caused by: java.lang.IllegalArgumentException: Partition key parts [segment, ts] does not match with partition values [cat1, 2024, 01, 01]. Check partition strategy.
    at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
    at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.getPartitionClause(QueryBasedDDLExecutor.java:191)
    at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.constructAddPartitions(QueryBasedDDLExecutor.java:164)
    at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.addPartitionsToTable(QueryBasedDDLExecutor.java:124)
    at org.apache.hudi.hive.HoodieHiveSyncClient.addPartitionsToTable(HoodieHiveSyncClient.java:118)
    at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:516)
    at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:470)
    ... 5 more {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8311
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.