opensearch-project / opensearch-project/sql

[FEATURE] Add new data type for text

Open
#1,038 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bi-tooling data-correctness enhancement Priority-High SQL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?

SQL plugin doesn't distinguish between text and keyword data types. OpenSearch supports aggregation on keywords and texts with fielddata and/or fields.

It is possible to aggregate on keyword or text (conditions apply)

opensearchsql> select sum(int0) from calcs GROUP BY str0;
fetched rows / total rows = 3/3
+-------------+
| sum(int0)   |
|-------------|
| 1           |
| 18          |
| 49          |
+-------------+

But impossible to aggregate on general text:

opensearchsql> select gender, count(firstname) from bank-with-null-values group by gender;
TransportError(500, 'SearchPhaseExecutionException', {'error': {'type': 'SearchPhaseExecutionException', 'reason': 'Error occurred in OpenSearch engine: all shards failed', 'details': 'Shard[0]: java.lang.IllegalArgumentException: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [gender] in order to load field data by uninverting the inverted index. Note that this can use significant memory.\n\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.'}, 'status': 503})

Existing mapping

JDBC type ExprCoreType OpenSearchDataType OpenSearch type
VARCHAR STRING OPENSEARCH_TEXT_KEYWORD keyword
VARCHAR STRING OPENSEARCH_TEXT text

See OpenSearch mapping samples available for aggregation:
https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/integ-test/src/test/resources/correctness/opensearch_dashboards_sample_data_flights.json#L25-L27
https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/integ-test/src/test/resources/correctness/opensearch_dashboards_sample_data_flights.json#L61-L69
https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/integ-test/src/test/resources/indexDefinitions/account_index_mapping.json#L12-L21
Not available for aggregation:
https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/integ-test/src/test/resources/indexDefinitions/bank_with_null_values_index_mapping.json#L16-L18

What solution would you like?

Have 2 different data types which are mapped to different JDBC/ODBC types.

JDBC type ExprCoreType OpenSearchDataType OpenSearch type
VARCHAR/CHAR STRING OPENSEARCH_KEYWORD keyword
text with fielddata
text with fields
LONGVARCHAR/TEXT TEXT OPENSEARCH_TEXT text without fielddata and fields

What alternatives have you considered?

N/A

Do you have any additional context?

Opened on behalf of @kylepbit
Ref:

  1. https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/sql-jdbc/src/main/java/org/opensearch/jdbc/types/OpenSearchType.java#L59-L61
  2. https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/opensearch/src/main/java/org/opensearch/sql/opensearch/data/type/OpenSearchDataType.java#L25-L46
  3. https://github.com/opensearch-project/sql/blob/b56edc7b693594b87fe7b0ffd635eff79148e179/core/src/main/java/org/opensearch/sql/data/type/ExprCoreType.java#L44-L47

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the referenced type definitions in sql-jdbc/src/main/java/org/opensearch/jdbc/types/OpenSearchType.java, opensearch/src/main/java/org/opensearch/sql/opensearch/data/type/OpenSearchDataType.java, and core/src/main/java/org/opensearch/sql/data/type/ExprCoreType.java. Compare these mappings with the linked index-definition samples and verify that keyword or fielddata-enabled text remains aggregatable while plain text maps to TEXT and is not treated the same way.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.