opensearch-project / opensearch-project/sql

Field with mapping of ip_range throwing SemanticCheckException - SQL Plugin

Open
#465 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature field types support
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Describe the bug

I am using AWS OpenSearch Service:

Version: OpenSearch 1.1 (latest)
Service software version: R20211203-P2 (latest)

I am trying to create a new index template, with a field mapping of "ip_range". I can see this is supported in Elastic Search - https://www.elastic.co/guide/en/elasticsearch/reference/current/range.html#ip-range and commonly used in the Elastic Common Standard (ECS) which is what I am basing my template off of (so assume this should be supported in OpenSearch).

Creating the index template is fine, and adding data to the index is fine, however, when I try to retrieve the data using SQL plugin, OpenSearch responds with an error.

To Reproduce

  1. create index template

Request:

curl -X PUT -k --user user:pass https://opensearch.url/_index_template/test-template  -H 'Content-Type: application/json' -d'
{
   "index_patterns":[
      "test-index"
   ],
   "data_stream":{
      
   },
   "priority":500,
   "template":{
      "aliases":{
         "test-template":{
            
         }
      },
      "settings":{
         "number_of_shards":1,
         "number_of_replicas":0
      },
      "mappings":{
         "properties":{
            "@timestamp":{
               "type":"date"
            },
            "myfield2":{
               "properties":{
                  "myfield1":{
                     "type":"ip_range"
                  }
               }
            },
            "myfield3":{
               "type":"ip_range"
            }
         }
      }
   }
}
' | jq
'

Response:

{
  "acknowledged": true
}
  1. Create data.json with the following
{"create":{ }}
{"myfield2":{"myfield1":"192.168.0.1/24"},"myfield3":"10.0.0.0/8","@timestamp":"2022-02-07T02:46:22.807Z"}
{"create":{ }}
{"myfield2":{"myfield1":"192.168.0.1/24"},"myfield3":"10.0.0.0/16","@timestamp":"2022-02-07T02:46:22.807Z"}

  1. Add it data to the index
curl -X PUT -k --user user:pass 'https://opensearch.url/test-index/_bulk?refresh&pretty' -H 'Content-Type: application/json' --data-binary @data.json

Output:

{
  "took" : 1449,
  "errors" : false,
  "items" : [
    {
      "create" : {
        "_index" : ".ds-test-index-000001",
        "_type" : "_doc",
        "_id" : "CDOMSH8BlRWPDhsq2_vv",
        "_version" : 1,
        "result" : "created",
        "forced_refresh" : true,
        "_shards" : {
          "total" : 1,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "create" : {
        "_index" : ".ds-test-index-000001",
        "_type" : "_doc",
        "_id" : "CTOMSH8BlRWPDhsq2_vv",
        "_version" : 1,
        "result" : "created",
        "forced_refresh" : true,
        "_shards" : {
          "total" : 1,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 1,
        "_primary_term" : 1,
        "status" : 201
      }
    }
  ]
}
  1. check the count went up
curl -X POST -k --user user:pass 'https://opensearch.url/_plugins/_sql?format=csv&pretty' -H 'Content-Type: application/json' -d'
{
  "query": "SELECT count(*) FROM test-index"
}
'

Output:

count(*)
2
  1. Try and select a specific record/all records
curl -X POST -k --user user:pass 'https://opensearch.url/_plugins/_sql?format=csv&pretty' -H 'Content-Type: application/json' -d'
{
  "query": "SELECT * FROM test-index LIMIT 1"
}
'

OR

curl -X POST -k --user user:pass 'https://opensearch.url/_plugins/_sql?format=csv&pretty' -H 'Content-Type: application/json' -d'
{
  "query": "SELECT myfield3 FROM test-index LIMIT 1"
}
'

Response:

{
  "type": "IllegalStateException",
  "reason": "No type found for field: myfield2.myfield1."
}

OR

{
  "error": {
    "reason": "Invalid SQL query",
    "details": "can't resolve Symbol(namespace=FIELD_NAME, name=myfield3) in type env",
    "type": "SemanticCheckException"
  },
  "status": 400
}
  1. Clean up the index
curl -X DELETE -k --user user:pass 'https://opensearch.url/_data_stream/test-index'

Expected behavior

I expect OpenSearch to respond back with the CIDR, and this CIDR should also be able to be used in range type queries.

Host/Environment (please complete the following information):

Version: OpenSearch 1.1 (latest)
Service software version: R20211203-P2 (latest)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure in the SQL plugin with the supplied index template, bulk data, and SELECT queries. Trace field and type resolution for the ip_range mappings, then verify that selecting the fields returns their CIDR values and supports range queries without SemanticCheckException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.