alibaba / alibaba/DataX

ElasticsearchWriter 字段type为date,format为“yyyy-MM-dd HH:mm:ss”,转换后格式不正确

Open
#1,020 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

由MySQL中集成数据到Elasticsearch中,Elasticsearch对应字段格式为:yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis,
DataxElasticsearchWriter 字段type为date,format为“yyyy-MM-dd HH:mm:ss”,执行后报错

[0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: {"message":"status:[400], error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [update_time] of type [date] in document with id '5'. Preview of field's value: '2021-04-19T10:43:36.000+08:00'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2021-04-19T10:43:36.000+08:00] with format [yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}","record":[{"byteSize":1,"index":0,"rawData":1,"type":"LONG"},{"byteSize":0,"index":1,"rawData":"","type":"STRING"},{"byteSize":9,"index":2,"rawData":"yicai.com","type":"STRING"},{"byteSize":8,"index":3,"rawData":1618800208000,"type":"DATE"},{"byteSize":5,"index":4,"rawData":"中国,上海","type":"STRING"},{"byteSize":1,"index":5,"rawData":5,"type":"LONG"},{"byteSize":5,"index":6,"rawData":"第一财经网","type":"STRING"},{"byteSize":2,"index":7,"rawData":"{}","type":"STRING"},{"byteSize":8,"index":8,"rawData":1618800216000,"type":"DATE"}],"type":"writer"}

跟踪代码后发现com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter类:
private String getDateStr(ESColumn esColumn, Column column) {
DateTime date = null;
DateTimeZone dtz = DateTimeZone.getDefault();
if (esColumn.getTimezone() != null) {
// 所有时区参考 http://www.joda.org/joda-time/timezones.html
dtz = DateTimeZone.forID(esColumn.getTimezone());
}
if (**column.getType() != Column.Type.DATE** && esColumn.getFormat() != null) {
DateTimeFormatter formatter = DateTimeFormat.forPattern(esColumn.getFormat());
date = formatter.withZone(dtz).parseDateTime(column.asString());
return **date.toString()**;
} else if (column.getType() == Column.Type.DATE) {
date = new DateTime(column.asLong(), dtz);
return date.toString();
} else {
return column.asString();
}
}
这部分代码对应ESWriter的日期类型转换没有起到作用。
以上加粗部分column.getType() != Column.Type.DATE 是不是应该改为column.getType() == Column.Type.DATE
date.toString() 是不是应该改为date.toString(formatter);
date.toString() :Output the date time in ISO8601 format (yyyy-MM-ddTHH:mm:ss.SSSZZ).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter, especially getDateStr, and reproduce the MySQL-to-Elasticsearch date conversion described in the issue. Check the handling of Column.Type.DATE, the configured format, and the emitted ISO8601 value; done means the configured date mapping accepts the converted value without the reported parsing error.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases, search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.