hive->mysql 提示“Column xxx cannot be null"
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
hive表为orc格式,表中部分字段为空值(`NULL`), 在配置文件中设置了`"nullFormat": "\\N"`,同步过程中仍然有部分记录出现`ERROR StdoutPluginCollector - 脏数据`提示
具体错误信息如下:
```
2019-04-11 16:31:42.634 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Column 'xx' cannot be null","record":
[{"byteSize":32,"index":0,"rawData":"","type":"STRING"},
{"byteSize":3,"index":1,"rawData":"xx","type":"STRING"},
{"byteSize":11,"index":2,"rawData":"xx","type":"STRING"},
{"byteSize":5,"index":3,"rawData":"xx","type":"STRING"},
{"byteSize":9,"index":4,"rawData":"xx","type":"STRING"},
{"byteSize":4,"index":5,"rawData":"xx","type":"STRING"},
{"byteSize":4,"index":6,"rawData":"xx","type":"STRING"},
{"byteSize":19,"index":7,"rawData":"2019-04-10 17:27:28","type":"STRING"},
{"byteSize":19,"index":8,"rawData":"2020-04-10 17:27:28","type":"STRING"},
{"byteSize":32,"index":9,"rawData":"xx","type":"STRING"},
{"byteSize":1,"index":10,"rawData":8,"type":"LONG"},
{"byteSize":3,"index":11,"rawData":"xx","type":"STRING"},
{"byteSize":6,"index":12,"rawData":100,"type":"LONG"},
{"byteSize":1,"index":13,"rawData":0,"type":"LONG"},
{"byteSize":3,"index":14,"rawData":"xx","type":"STRING"},
{"byteSize":19,"index":15,"rawData":"2019-04-10 17:27:28","type":"STRING"},
{"byteSize":7,"index":16,"rawData":"natural","type":"STRING"},
{"byteSize":0,"index":17,"type":"STRING"},
{"byteSize":0,"index":18,"type":"STRING"},
{"byteSize":19,"index":19,"rawData":"2019-04-10 17:06:03","type":"STRING"},
{"byteSize":1,"index":20,"rawData":0,"type":"LONG"},
{"byteSize":0,"index":21,"rawData":"","type":"STRING"},
{"byteSize":1,"index":22,"rawData":1,"type":"LONG"},
{"byteSize":10,"index":23,"rawData":"2019-04-10","type":"STRING"},
{"byteSize":19,"index":24,"rawData":"2019-04-11 01:15:10","type":"STRING"}],
"type":"writer"}
```
提示不能为空的字段index是17
并不是所有存在空值字段的记录都不能正常同步,只有一部分存在空值问题。
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the hive-mysql synchronization path and the configuration handling for nullFormat, using the reported ORC records with NULL values and the writer error showing index 17. Reproduce the difference between records that synchronize successfully and those reported as dirty data, then verify that the affected NULL field is handled consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100