opensearch-project / opensearch-project/sql

[FEATURE] Support non-keyword fields for `text`

Open
#1,112 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement SQL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?

Search doesn't work properly for fields of non-keyword data type.
Mapping

{
  "mappings": {
    "properties": {
      "numbers": {
        "type": "text",
        "fields": {
          "values": {
            "type": "integer"
          }
        }
      }
    }
  }
}

Create new index:

$ curl -s -H 'Content-Type: application/json' -XPUT "http://localhost:9200/tmp?pretty" -d '{"mappings": {"properties": {"numbers": {"type": "text", "fields": {"values": {"type": "integer"}}}}}}'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "tmp"
}

Add a doc:

$ curl -s -H 'Content-Type: application/json' -XPOST "http://localhost:9200/tmp/_doc?pretty" -d '{"numbers": [12, 20, 30]}'

Validate data:

$ curl -s -XGET "http://localhost:9200/tmp/_search?pretty" | jq .hits.hits
[
  {
    "_index": "tmp",
    "_id": "wNh6soQBeC3wK8rxBHNm",
    "_score": 1,
    "_source": {
      "numbers": [
        12,
        20,
        30
      ]
    }
  }
]
$ curl -s -H 'Content-Type: application/json' -XGET "http://localhost:9200/tmp/_search?pretty" -d '{"query": {"match_all": {}}, "fields": ["numbers", "numbers.values"], "_source": false}' | jq .hits.hits
[
  {
    "_index": "tmp",
    "_id": "wNh6soQBeC3wK8rxBHNm",
    "_score": 1,
    "fields": {
      "numbers.values": [
        12,
        20,
        30
      ],
      "numbers": [
        "12",
        "20",
        "30"
      ]
    }
  }
]

Try an SQL query:

opensearchsql> select numbers from tmp;
fetched rows / total rows = 1/1
+-----------+
| numbers   |
|-----------|
| 12        |
+-----------+
opensearchsql> select numbers.values from tmp;
{'reason': 'Invalid SQL query', 'details': "can't resolve Symbol(namespace=FIELD_NAME, name=numbers.values) in type env", 'type': 'SemanticCheckException'}
opensearchsql> select * from tmp where numbers.values = 20;
{'reason': 'Invalid SQL query', 'details': "can't resolve Symbol(namespace=FIELD_NAME, name=numbers.values) in type env", 'type': 'SemanticCheckException'}
$ curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"query": "select numbers from tmp;"}'
{
  "schema": [
    {
      "name": "numbers",
      "type": "text"
    }
  ],
  "datarows": [
    [
      "12"
    ]
  ],
  "total": 1,
  "size": 1,
  "status": 200
}

What solution would you like?

Listed above queries should return something like this:

  • 12 20 30
  • 12, 20, 30
  • "12 20 30"
  • [12, 20, 30]
  • ["12", "20", "30"]

What alternatives have you considered?

Add limitation section to documentation and return error on querying such kind of fields (aka columns).

Do you have any additional context?

Related to #794, #1038 and #1113.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no source files, entry points, or tests; begin by tracing SQL field resolution for mapped text subfields and review related issues #794, #1038, and #1113. Done means the listed queries resolve non-keyword fields and return the requested scalar or array values, with coverage for the mapping shown.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.