alibaba / alibaba/DataX

从mysql 同步数据到 elasticsearch 时,只同步更新 elasticsearch 部分字段,不更新的字段不覆盖,可以吗

Open
#1,213 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

{
"job": {
"setting": {
"speed": {
"channel": 2,
"record": -1,
"byte": -1,
"batchSize": 2048
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "xxx",
"password": "xxx",
"connection": [
{
"querySql": [
"SELECT t.id as _id, t.id as id, td.delegation as delegation FROM uat_partner.t_house_info t left join uat_partner.t_house_supporting_info ts on t.id = ts.house_id left join uat_partner.t_house_delegation_info td on t.id = td.house_id left join uat_hb_manager.sys_office ks on t.know_office_id = ks.id where t.id = 2111010924279766254 order by t.update_date desc;"
],
"jdbcUrl": [
"jdbc:mysql://xxx.xx.xxx.xxx:3306"
]
}
]
}
},
"writer": {
"name": "elasticsearchwriter",
"parameter": {
"endpoint": "http://xxx.xx.xx.xx:9200",
"accessId": "xxx",
"accessKey": "xxx",
"index": "in_index",
"indexType": "default",
"type": "_doc",
"cleanup": false,
"dynamic": false,
"batchSize": 1000,
"trySize": 30,
"multiThread": true,
"column": [
{
"name": "_id",
"type": "id"
},
{
"name": "id",
"type": "long"
},
{
"name": "delegation",
"type": "long"
}
]
}
}
}
]
}
}

我只想同步更新 ES 中的 delegation 字段,其它字段的值不更新,有什么方式吗

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied DataX job configuration, especially the mysqlreader and elasticsearchwriter entries and the writer's column mapping. Investigate how elasticsearchwriter handles records when only delegation is selected, then define and verify behavior where unspecified Elasticsearch fields remain unchanged while delegation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java, mysql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.