elasticsearchwriter: the problem that datetime type of Synchronizing MySQL‘s data to ES
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
if MySQL's type is datetime,Synchronizing the data to ES,you will find the data of MySQL is null becoming not null。And the time to be local timeStamp.We can fix this bug like this:
(1)find the package "com.alibaba.datax.plugin.writer.elasticsearchwriter" ,then find the method of "getDateStr" ,which in the class of "ESWriter".
(2)change the method as follows:
/**
* ES time parse
* @param esColumn
* @param column
* @return
*/
private String getDateStr(ESColumn esColumn, Column column) {
DateTime date = null;
DateTimeZone dtz = DateTimeZone.getDefault();
if (esColumn.getTimezone() != null) {
dtz = DateTimeZone.forID(esColumn.getTimezone());
}
if (column.getType() != Column.Type.DATE && esColumn.getFormat() != null) {
DateTimeFormatter formatter = DateTimeFormat.forPattern(esColumn.getFormat());
date = formatter.withZone(dtz).parseDateTime(column.asString());
return date.toString();
} else if (column.getType() == Column.Type.DATE) {
//time type of es will executing this
//add column.asLong() not null
if(column.asLong() != null){
date = new DateTime(column.asLong(), dtz);
//if you change the time format of fields like "yyyy-MM-dd HH:mm:ss",not ""yyyy-MM-dd'T'HH:mm:ss" +8:00",you must create the mapping of ES time field like this manually
return date.toString("yyyy-MM-dd HH:mm:ss");
}
//if column.asLong() is null, Output as null
return null;
} else {
return column.asString();
}
}
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the com.alibaba.datax.plugin.writer.elasticsearchwriter package and inspect getDateStr in ESWriter, along with the existing ESColumn and Column date handling. Reproduce the MySQL datetime case and verify that null values remain null and non-null values use the requested timezone and format; the issue provides the proposed behavior but names no test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100