alibaba / alibaba/DataX

datax同步null值数据到clickhouse异常问题

Open
#1,671 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

从hive往clickhouse同步数据,执行数据的时候,有部分数据的create_time为null,数据类型为DateTime

2023-02-08 11:23:27.474 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:ClickHouse exception, code: 27, host: hadoop10086, port: 8082; Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected '\t' before: '餐饮\t中餐厅\t中餐厅\t其他\t辽宁省\t本溪市\t\t本溪满族自治县\t202301\t\n92cabd0939c043c6b2ec775c7c461744\t\t\t2750731168541316837\tPO': (at row 1)
Row 1:
Column 0, name: id, type: String, parsed text: "10ass92cabd072f454"
Column 1, name: city_id, type: String, parsed text:
Column 2, name: category_id, type: String, parsed text:
Column 3, name: grd, type: String, parsed text: "100021235123121"
Column 4, name: location, type: String, parsed text: "124.13025 41.2943"
Column 5, name: address, type: String, parsed text: "阳光花园小区西区3号门旁"
Column 6, name: name, type: String, parsed text: "福禾顺"
Column 7, name: create_time, type: DateTime, parsed text: "N"ERROR: DateTime must be in YYYY-MM-DD hh:mm:ss or NNNNNNNNNN (unix timestamp, exactly 10 digits) format.
Code: 117, e.displayText() = DB::Exception: Unexpected NULL value of not Nullable type DateTime (version 20.11.3.3 (official build))

如果字段是string类型,可以提前将数据转换为空字符串。
而时间类型的字段数据为null,应该需要怎么赋值,再同步。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported Hive-to-ClickHouse synchronization case and its DateTime error log. Define the expected representation for a null create_time, then verify that a representative row synchronizes without the reported parse and non-Nullable errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.