datax同步null值数据到clickhouse异常问题
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
从hive往clickhouse同步数据,执行数据的时候,有部分数据的create_time为null,数据类型为DateTime
2023-02-08 11:23:27.474 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:ClickHouse exception, code: 27, host: hadoop10086, port: 8082; Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected '\t' before: '餐饮\t中餐厅\t中餐厅\t其他\t辽宁省\t本溪市\t\t本溪满族自治县\t202301\t\n92cabd0939c043c6b2ec775c7c461744\t\t\t2750731168541316837\tPO': (at row 1)
Row 1:
Column 0, name: id, type: String, parsed text: "10ass92cabd072f454"
Column 1, name: city_id, type: String, parsed text:
Column 2, name: category_id, type: String, parsed text:
Column 3, name: grd, type: String, parsed text: "100021235123121"
Column 4, name: location, type: String, parsed text: "124.13025 41.2943"
Column 5, name: address, type: String, parsed text: "阳光花园小区西区3号门旁"
Column 6, name: name, type: String, parsed text: "福禾顺"
Column 7, name: create_time, type: DateTime, parsed text: "N"ERROR: DateTime must be in YYYY-MM-DD hh:mm:ss or NNNNNNNNNN (unix timestamp, exactly 10 digits) format.
Code: 117, e.displayText() = DB::Exception: Unexpected NULL value of not Nullable type DateTime (version 20.11.3.3 (official build))
如果字段是string类型,可以提前将数据转换为空字符串。
而时间类型的字段数据为null,应该需要怎么赋值,再同步。
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported Hive-to-ClickHouse synchronization case and its DateTime error log. Define the expected representation for a null create_time, then verify that a representative row synchronizes without the reported parse and non-Nullable errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100