alibaba / alibaba/DataX

我下载过了,并没有什么用,没有文档没有源码。出问题也无法定位到问题。建议别用。> es动态column 有实现案例吗

Open
#1,398 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

> es动态column 有实现案例吗

> > > > > > 使用datax elasticsearchwriter往 es 7.9.2 版本写入数据,column配置没有生效
> > > > > > elasticsearch版本7.9.2
> > > > > > datax配置:
> > > > > > "writer": {
> > > > > > "name": "elasticsearchwriter",
> > > > > > "parameter": {
> > > > > > "endpoint": "http://XXX:9200",
> > > > > > "accessId": "xxxx",
> > > > > > "accessKey": "xxxx",
> > > > > > "index": "XXXX",
> > > > > > "type": "_doc",
> > > > > > "cleanup": true,
> > > > > > "settings": {"index": {"number_of_shards": 1, "number_of_replicas": 0}},
> > > > > > "column": [
> > > > > > {"name": "company_id", "type": "integer"},
> > > > > > { "name": "company_nm","type": "text" },
> > > > > > { "name": "is_list","type": "integer" },
> > > > > > { "name": "list_type","type": "integer" },
> > > > > > { "name": "is_new_otc","type": "integer" },
> > > > > > { "name": "is_bond","type": "integer" },
> > > > > > { "name": "is_finance","type": "integer" },
> > > > > > { "name": "fin_type","type": "integer" },
> > > > > > { "name": "fin_type_supp","type": "text" },
> > > > > > { "name": "is_itfin","type": "integer" },
> > > > > > { "name": "is_del","type": "integer" },
> > > > > > { "name": "create_dt","type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},
> > > > > > { "name": "updt_dt","type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"}
> > > > > > ],
> > > > > > "discovery": false,
> > > > > > "dynamic": false,
> > > > > > "batchSize": 1000
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > > 最终生成的映射中column配置都没有生效,全量使用的是es默认mapping,特别是日期格式
> > > > >
> > > > >
> > > > > es客户端过时了,创建不了索引,得用es7的客户端改源码
> > > >
> > > >
> > > > 我同意你说的,目前提供的elasticsearchwriter还是旧版的,准备尝试使用elasticsearch7.x的客户端来改造writer
> > > > 链接:https://pan.baidu.com/s/1Wt5M10GlamNyI8MaqYZ3nQ
> > > > 提取码:137e,官方版本比较老,我这个已经升级,可以直接使用
> >
> >
> > 谢谢~
>
> 链接:https://pan.baidu.com/s/1PNIpPBCyKYTYpofFiwq3Wg 提取码:uk9z,之前的还是老版本,新版本地址是这个,请使用,有问题反馈

您好,您改造了elasticsearchwriter 是吗? 支持elasticsearch 7.x ? 方便提供下源码吗?

_Originally posted by @yunf17 in https://github.com/alibaba/DataX/issues/867#issuecomment-1120129113_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the elasticsearchwriter implementation and its writer configuration, focusing on index creation, column mappings, dynamic settings, and the Elasticsearch client version. Reproduce the configuration against Elasticsearch 7.9.2; done means the writer creates the index with the requested mappings, including date formats, rather than relying on the default mapping.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.