opensearch-project / opensearch-project/sql
Field with mapping of ip_range throwing SemanticCheckException - SQL Plugin
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Describe the bug
I am using AWS OpenSearch Service:
Version: OpenSearch 1.1 (latest)
Service software version: R20211203-P2 (latest)
I am trying to create a new index template, with a field mapping of "ip_range". I can see this is supported in Elastic Search - https://www.elastic.co/guide/en/elasticsearch/reference/current/range.html#ip-range and commonly used in the Elastic Common Standard (ECS) which is what I am basing my template off of (so assume this should be supported in OpenSearch).
Creating the index template is fine, and adding data to the index is fine, however, when I try to retrieve the data using SQL plugin, OpenSearch responds with an error.
To Reproduce
- create index template
Request:
curl -X PUT -k --user user:pass https://opensearch.url/_index_template/test-template -H 'Content-Type: application/json' -d'
{
"index_patterns":[
"test-index"
],
"data_stream":{
},
"priority":500,
"template":{
"aliases":{
"test-template":{
}
},
"settings":{
"number_of_shards":1,
"number_of_replicas":0
},
"mappings":{
"properties":{
"@timestamp":{
"type":"date"
},
"myfield2":{
"properties":{
"myfield1":{
"type":"ip_range"
}
}
},
"myfield3":{
"type":"ip_range"
}
}
}
}
}
' | jq
'
Response:
{
"acknowledged": true
}
- Create data.json with the following
{"create":{ }}
{"myfield2":{"myfield1":"192.168.0.1/24"},"myfield3":"10.0.0.0/8","@timestamp":"2022-02-07T02:46:22.807Z"}
{"create":{ }}
{"myfield2":{"myfield1":"192.168.0.1/24"},"myfield3":"10.0.0.0/16","@timestamp":"2022-02-07T02:46:22.807Z"}
- Add it data to the index
curl -X PUT -k --user user:pass 'https://opensearch.url/test-index/_bulk?refresh&pretty' -H 'Content-Type: application/json' --data-binary @data.json
Output:
{
"took" : 1449,
"errors" : false,
"items" : [
{
"create" : {
"_index" : ".ds-test-index-000001",
"_type" : "_doc",
"_id" : "CDOMSH8BlRWPDhsq2_vv",
"_version" : 1,
"result" : "created",
"forced_refresh" : true,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1,
"status" : 201
}
},
{
"create" : {
"_index" : ".ds-test-index-000001",
"_type" : "_doc",
"_id" : "CTOMSH8BlRWPDhsq2_vv",
"_version" : 1,
"result" : "created",
"forced_refresh" : true,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1,
"status" : 201
}
}
]
}
- check the count went up
curl -X POST -k --user user:pass 'https://opensearch.url/_plugins/_sql?format=csv&pretty' -H 'Content-Type: application/json' -d'
{
"query": "SELECT count(*) FROM test-index"
}
'
Output:
count(*)
2
- Try and select a specific record/all records
curl -X POST -k --user user:pass 'https://opensearch.url/_plugins/_sql?format=csv&pretty' -H 'Content-Type: application/json' -d'
{
"query": "SELECT * FROM test-index LIMIT 1"
}
'
OR
curl -X POST -k --user user:pass 'https://opensearch.url/_plugins/_sql?format=csv&pretty' -H 'Content-Type: application/json' -d'
{
"query": "SELECT myfield3 FROM test-index LIMIT 1"
}
'
Response:
{
"type": "IllegalStateException",
"reason": "No type found for field: myfield2.myfield1."
}
OR
{
"error": {
"reason": "Invalid SQL query",
"details": "can't resolve Symbol(namespace=FIELD_NAME, name=myfield3) in type env",
"type": "SemanticCheckException"
},
"status": 400
}
- Clean up the index
curl -X DELETE -k --user user:pass 'https://opensearch.url/_data_stream/test-index'
Expected behavior
I expect OpenSearch to respond back with the CIDR, and this CIDR should also be able to be used in range type queries.
Host/Environment (please complete the following information):
Version: OpenSearch 1.1 (latest)
Service software version: R20211203-P2 (latest)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure in the SQL plugin with the supplied index template, bulk data, and SELECT queries. Trace field and type resolution for the ip_range mappings, then verify that selecting the fields returns their CIDR values and supports range queries without SemanticCheckException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100