apache / apache/hudi

[SUPPORT]flink-sql write hudi use TIMESTAMP, when hive query, it get time+8h question, use TIMESTAMP_LTZ, the hive schema is bigint but timestamp

Open
#9,864 13 comments 0 reactions 0 assignees View on GitHub
area:schema engine:flink priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**when I use sql with timestamp:**

**CREATE TABLE dwd_ratio_path_company(
id DECIMAL(20, 0),
company_id BIGINT,
shareholder_id STRING,
shareholder_entity_type SMALLINT,
shareholder_name_id BIGINT,
investment_ratio_total DECIMAL(24, 12),
is_controller SMALLINT,
is_ultimate SMALLINT,
is_big_shareholder SMALLINT,
is_controlling_shareholder SMALLINT,
equity_holding_path STRING,
create_time TIMESTAMP(3),
update_time TIMESTAMP(3),
is_deleted SMALLINT,
op_ts TIMESTAMP(3),
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hudi',
'path' = 'obs://hadoop-obs/hudi_ods/ratio_path_company015',
'table.type' = 'MERGE_ON_READ',
-- index
'index.type' = 'BUCKET',
'hoodie.bucket.index.num.buckets' = '128',
-- write
'write.tasks' = '4',
'write.task.max.size' = '512',
'write.batch.size' = '8',
'write.log_block.size' = '64',
'write.precombine' = 'true',
'write.precombine.field' = 'op_ts',
-- compaction
'compaction.async.enabled' = 'true',
'compaction.delta_commits' = '3',
-- clean
'clean.retain_commits' = '0',
-- hive
'hive_sync.enabled' = 'true',
'hive_sync.db' = 'hudi_ods',
'hive_sync.table' = 'ratio_path_company015',
'hive_sync.metastore.uris' = 'thrift://10.99.202.153:9083',
'hive_sync.mode' = 'hms',
'hive_sync.support_timestamp' = 'true'
);
insert into dwd_ratio_path_company select * from ods_ratio_path_company;**

it appear:
image
image

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the SQL table definition and INSERT statement in the report, focusing on TIMESTAMP(3), TIMESTAMP_LTZ, Hudi, and the hive_sync.support_timestamp setting. Compare the resulting Hive schema and queried values shown in the screenshots; the issue is resolved when the timestamp type and timezone behavior are explained or corrected consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.