alibaba / alibaba/DataX

elasticsearchwriter对 es7.9.2 版本column映射不生效

Open
#867 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

使用datax elasticsearchwriter往 es 7.9.2 版本写入数据,column配置没有生效
elasticsearch版本7.9.2
datax配置:
"writer": {
"name": "elasticsearchwriter",
"parameter": {
"endpoint": "http://XXX:9200",
"accessId": "xxxx",
"accessKey": "xxxx",
"index": "XXXX",
"type": "_doc",
"cleanup": true,
"settings": {"index": {"number_of_shards": 1, "number_of_replicas": 0}},
"column": [
{"name": "company_id", "type": "integer"},
{ "name": "company_nm","type": "text" },
{ "name": "is_list","type": "integer" },
{ "name": "list_type","type": "integer" },
{ "name": "is_new_otc","type": "integer" },
{ "name": "is_bond","type": "integer" },
{ "name": "is_finance","type": "integer" },
{ "name": "fin_type","type": "integer" },
{ "name": "fin_type_supp","type": "text" },
{ "name": "is_itfin","type": "integer" },
{ "name": "is_del","type": "integer" },
{ "name": "create_dt","type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},
{ "name": "updt_dt","type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"}
],
"discovery": false,
"dynamic": false,
"batchSize": 1000
}
}
}

最终生成的映射中column配置都没有生效,全量使用的是es默认mapping,特别是日期格式

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the elasticsearchwriter entry point and reproduce the issue with the supplied Elasticsearch 7.9.2 configuration. Compare the generated index mapping with the column definitions, especially the date format and integer/text types; done means the configured mapping is applied instead of Elasticsearch defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.