alibaba / alibaba/DataX

hive->mysql 提示“Column xxx cannot be null"

Open
#318 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

hive表为orc格式,表中部分字段为空值(`NULL`), 在配置文件中设置了`"nullFormat": "\\N"`,同步过程中仍然有部分记录出现`ERROR StdoutPluginCollector - 脏数据`提示
具体错误信息如下:
```
2019-04-11 16:31:42.634 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Column 'xx' cannot be null","record":
[{"byteSize":32,"index":0,"rawData":"","type":"STRING"},
{"byteSize":3,"index":1,"rawData":"xx","type":"STRING"},
{"byteSize":11,"index":2,"rawData":"xx","type":"STRING"},
{"byteSize":5,"index":3,"rawData":"xx","type":"STRING"},
{"byteSize":9,"index":4,"rawData":"xx","type":"STRING"},
{"byteSize":4,"index":5,"rawData":"xx","type":"STRING"},
{"byteSize":4,"index":6,"rawData":"xx","type":"STRING"},
{"byteSize":19,"index":7,"rawData":"2019-04-10 17:27:28","type":"STRING"},
{"byteSize":19,"index":8,"rawData":"2020-04-10 17:27:28","type":"STRING"},
{"byteSize":32,"index":9,"rawData":"xx","type":"STRING"},
{"byteSize":1,"index":10,"rawData":8,"type":"LONG"},
{"byteSize":3,"index":11,"rawData":"xx","type":"STRING"},
{"byteSize":6,"index":12,"rawData":100,"type":"LONG"},
{"byteSize":1,"index":13,"rawData":0,"type":"LONG"},
{"byteSize":3,"index":14,"rawData":"xx","type":"STRING"},
{"byteSize":19,"index":15,"rawData":"2019-04-10 17:27:28","type":"STRING"},
{"byteSize":7,"index":16,"rawData":"natural","type":"STRING"},
{"byteSize":0,"index":17,"type":"STRING"},
{"byteSize":0,"index":18,"type":"STRING"},
{"byteSize":19,"index":19,"rawData":"2019-04-10 17:06:03","type":"STRING"},
{"byteSize":1,"index":20,"rawData":0,"type":"LONG"},
{"byteSize":0,"index":21,"rawData":"","type":"STRING"},
{"byteSize":1,"index":22,"rawData":1,"type":"LONG"},
{"byteSize":10,"index":23,"rawData":"2019-04-10","type":"STRING"},
{"byteSize":19,"index":24,"rawData":"2019-04-11 01:15:10","type":"STRING"}],
"type":"writer"}
```
提示不能为空的字段index是17
并不是所有存在空值字段的记录都不能正常同步,只有一部分存在空值问题。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the hive-mysql synchronization path and the configuration handling for nullFormat, using the reported ORC records with NULL values and the writer error showing index 17. Reproduce the difference between records that synchronize successfully and those reported as dirty data, then verify that the affected NULL field is handled consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.