alibaba / alibaba/DataX

geo_point类型转换失败

Open
#283 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

# 环境/配置
* es: 6.3.2
* "cleanup": true

# 情况一:三个字段,类型分别为date, geo_point, date时

```sql
select
current_date as date,
'23.0391693115234,113.137992858887' as location,
'2012-12-12' as name
```

```json
"column": [
{
"name": "date",
"type": "date"
},
{
"name": "location",
"type": "geo_point"
},
{
"name": "name",
"type": "date"
}
]
```

写入成功后,mapping没问题
```json
{
"ar_test_geo": {
"aliases": {},
"mappings": {
"_doc": {
"properties": {
"date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd"
},
"location": {
"type": "geo_point"
},
"name": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd"
}
}
}
}
}
```

# 情况二:三个字段,类型分别为date, geo_point, string时

```sql
select
current_date as date,
'23.0391693115234,113.137992858887' as location,
'aa' as name
```

```json
"column": [
{
"name": "date",
"type": "date"
},
{
"name": "location",
"type": "geo_point"
},
{
"name": "name",
"type": "string"
}
]
```

写入成功后,geo_point转成了text,为啥?
```json
{
"ar_test_geo": {
"aliases": {},
"mappings": {
"_doc": {
"properties": {
"date": {
"type": "date"
},
"location": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
```

# 暂时解决方法
只能通过writer的settings配置来指定mapping了,这个就和column配置冗余了,同时需要管理两份元数据。

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce both SQL cases with Elasticsearch 6.3.2, comparing the column configuration with the writer settings that define the mapping. Inspect the writer's mapping-generation entry point and determine why a geo_point becomes text when followed by a string column. Done means the generated mapping preserves geo_point without requiring duplicate mapping metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.