oracle 同步数据到 elasticsearch,date格式出错
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
同步的sql:日期进行了格式化处理
select a.id, to_char(sysdate, 'yyyy-mm-dd hh24:mi:ss') AS aae036, a.aac001, a.aac002, a.aac003, a.yac073, a.aab001 from a
datax的column json:
`"column": [
{"name":"id","type":"id"},
{"name":"city","type": "keyword"},
{"name":"aae036","type": "date","format":"yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd||epoch_millis"},
{"name":"aac002","type": "keyword","ignore_above": 256},
{"name":"aac003","type": "keyword","ignore_above": 256},
{"name":"aac004","type": "keyword","ignore_above": 256},
{"name":"yac005","type": "keyword","ignore_above": 256},
{"name":"yac150","type": "keyword","ignore_above": 256},
{"name":"yac151","type": "keyword","ignore_above": 256}
]`
在进行同步的时候,依旧出现了错误
elasticsearch日志错误:
[2018-07-11T18:19:30,914][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [node-1] failed to put mappings on indices [[[server_1004_v2/qJCsg7GCSJmz_10sCx3ABw]]], type [server_1004_v2]
java.lang.IllegalArgumentException: Mapper for [aae036] conflicts with existing mapping in other types:
[mapper [aae036] has different [format] values]
at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:130) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.FieldTypeLookup.copyAndAddAll(FieldTypeLookup.java:94) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:400) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
index的mapping:
`{
"server_1004_v2": {
"mappings": {
"server_1004_v2": {
"properties": {
"aac002": {
"type": "keyword",
"ignore_above": 256
},
"aac003": {
"type": "keyword",
"ignore_above": 256
},
"aae036": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||epoch_millis"
},
"city": {
"type": "keyword",
"ignore_above": 256
},
"yac005": {
"type": "keyword",
"ignore_above": 256
},
"ykc177": {
"type": "double"
}
}
}
}
}
}`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the DataX synchronization with the SQL, column configuration, and existing Elasticsearch mapping shown in the issue. Inspect the Oracle-to-Elasticsearch synchronization entry point and compare how the date format is generated with the existing mapping; done means the mapping can be applied without the reported format conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100