ElasticsearchWriter 字段type为date,format为“yyyy-MM-dd HH:mm:ss”,转换后格式不正确
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
由MySQL中集成数据到Elasticsearch中,Elasticsearch对应字段格式为:yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis,
DataxElasticsearchWriter 字段type为date,format为“yyyy-MM-dd HH:mm:ss”,执行后报错
[0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: {"message":"status:[400], error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [update_time] of type [date] in document with id '5'. Preview of field's value: '2021-04-19T10:43:36.000+08:00'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2021-04-19T10:43:36.000+08:00] with format [yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}","record":[{"byteSize":1,"index":0,"rawData":1,"type":"LONG"},{"byteSize":0,"index":1,"rawData":"","type":"STRING"},{"byteSize":9,"index":2,"rawData":"yicai.com","type":"STRING"},{"byteSize":8,"index":3,"rawData":1618800208000,"type":"DATE"},{"byteSize":5,"index":4,"rawData":"中国,上海","type":"STRING"},{"byteSize":1,"index":5,"rawData":5,"type":"LONG"},{"byteSize":5,"index":6,"rawData":"第一财经网","type":"STRING"},{"byteSize":2,"index":7,"rawData":"{}","type":"STRING"},{"byteSize":8,"index":8,"rawData":1618800216000,"type":"DATE"}],"type":"writer"}
跟踪代码后发现com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter类:
private String getDateStr(ESColumn esColumn, Column column) {
DateTime date = null;
DateTimeZone dtz = DateTimeZone.getDefault();
if (esColumn.getTimezone() != null) {
// 所有时区参考 http://www.joda.org/joda-time/timezones.html
dtz = DateTimeZone.forID(esColumn.getTimezone());
}
if (**column.getType() != Column.Type.DATE** && esColumn.getFormat() != null) {
DateTimeFormatter formatter = DateTimeFormat.forPattern(esColumn.getFormat());
date = formatter.withZone(dtz).parseDateTime(column.asString());
return **date.toString()**;
} else if (column.getType() == Column.Type.DATE) {
date = new DateTime(column.asLong(), dtz);
return date.toString();
} else {
return column.asString();
}
}
这部分代码对应ESWriter的日期类型转换没有起到作用。
以上加粗部分column.getType() != Column.Type.DATE 是不是应该改为column.getType() == Column.Type.DATE
date.toString() 是不是应该改为date.toString(formatter);
date.toString() :Output the date time in ISO8601 format (yyyy-MM-ddTHH:mm:ss.SSSZZ).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter, especially getDateStr, and reproduce the MySQL-to-Elasticsearch date conversion described in the issue. Check the handling of Column.Type.DATE, the configured format, and the emitted ISO8601 value; done means the configured date mapping accepts the converted value without the reported parsing error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100