alibaba / alibaba/DataX

oracle 同步数据到 elasticsearch,date格式出错

Open
#135 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

同步的sql:日期进行了格式化处理
select a.id, to_char(sysdate, 'yyyy-mm-dd hh24:mi:ss') AS aae036, a.aac001, a.aac002, a.aac003, a.yac073, a.aab001 from a

datax的column json:
`"column": [
{"name":"id","type":"id"},
{"name":"city","type": "keyword"},
{"name":"aae036","type": "date","format":"yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd||epoch_millis"},
{"name":"aac002","type": "keyword","ignore_above": 256},
{"name":"aac003","type": "keyword","ignore_above": 256},
{"name":"aac004","type": "keyword","ignore_above": 256},
{"name":"yac005","type": "keyword","ignore_above": 256},
{"name":"yac150","type": "keyword","ignore_above": 256},
{"name":"yac151","type": "keyword","ignore_above": 256}
]`

在进行同步的时候,依旧出现了错误

elasticsearch日志错误:
[2018-07-11T18:19:30,914][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [node-1] failed to put mappings on indices [[[server_1004_v2/qJCsg7GCSJmz_10sCx3ABw]]], type [server_1004_v2]
java.lang.IllegalArgumentException: Mapper for [aae036] conflicts with existing mapping in other types:
[mapper [aae036] has different [format] values]
at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:130) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.FieldTypeLookup.copyAndAddAll(FieldTypeLookup.java:94) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:400) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

index的mapping:
`{
"server_1004_v2": {
"mappings": {
"server_1004_v2": {
"properties": {
"aac002": {
"type": "keyword",
"ignore_above": 256
},
"aac003": {
"type": "keyword",
"ignore_above": 256
},
"aae036": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||epoch_millis"
},
"city": {
"type": "keyword",
"ignore_above": 256
},
"yac005": {
"type": "keyword",
"ignore_above": 256
},
"ykc177": {
"type": "double"
}
}
}
}
}
}`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the DataX synchronization with the SQL, column configuration, and existing Elasticsearch mapping shown in the issue. Inspect the Oracle-to-Elasticsearch synchronization entry point and compare how the date format is generated with the existing mapping; done means the mapping can be applied without the reported format conflict.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.