alibaba / alibaba/DataX

elasticsearchwriter: the problem that datetime type of Synchronizing MySQL‘s data to ES

Open
#933 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

if MySQL's type is datetime,Synchronizing the data to ES,you will find the data of MySQL is null becoming not null。And the time to be local timeStamp.We can fix this bug like this:
(1)find the package "com.alibaba.datax.plugin.writer.elasticsearchwriter" ,then find the method of "getDateStr" ,which in the class of "ESWriter".
(2)change the method as follows:

/**
* ES time parse
* @param esColumn
* @param column
* @return
*/
private String getDateStr(ESColumn esColumn, Column column) {
DateTime date = null;
DateTimeZone dtz = DateTimeZone.getDefault();

if (esColumn.getTimezone() != null) {

dtz = DateTimeZone.forID(esColumn.getTimezone());
}
if (column.getType() != Column.Type.DATE && esColumn.getFormat() != null) {

DateTimeFormatter formatter = DateTimeFormat.forPattern(esColumn.getFormat());
date = formatter.withZone(dtz).parseDateTime(column.asString());

return date.toString();
} else if (column.getType() == Column.Type.DATE) {
//time type of es will executing this
//add column.asLong() not null
if(column.asLong() != null){
date = new DateTime(column.asLong(), dtz);
//if you change the time format of fields like "yyyy-MM-dd HH:mm:ss",not ""yyyy-MM-dd'T'HH:mm:ss" +8:00",you must create the mapping of ES time field like this manually
return date.toString("yyyy-MM-dd HH:mm:ss");
}
//if column.asLong() is null, Output as null
return null;
} else {
return column.asString();
}
}

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the com.alibaba.datax.plugin.writer.elasticsearchwriter package and inspect getDateStr in ESWriter, along with the existing ESColumn and Column date handling. Reproduce the MySQL datetime case and verify that null values remain null and non-null values use the requested timezone and format; the issue provides the proposed behavior but names no test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.