opensearch-project / opensearch-project/sql

[BUG] SQL confusing type conversion for TIMESTAMP when using join

Open
#3,204 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug legacy SQL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Describe the bug

So given 2 indexes, with a matching field (to be used in a JOIN) and a timestamp field. When querying without a join, a WHERE clause like WHERE t.timestamp > timestamp('2001-05-07 00:00:00') will work and WHERE t.timestamp > 0 won't work since it's a TIMESTAMP field, that's what opensearch tell you

{\n  \"error\": {\n    \"reason\": \"Invalid SQL query\",\n    \"details\": \"> function expected {[BYTE,BYTE],[SHORT,SHORT],[INTEGER,INTEGER],[LONG,LONG],[FLOAT,FLOAT],[DOUBLE,DOUBLE],[STRING,STRING],[BOOLEAN,BOOLEAN],[DATE,DATE],[TIME,TIME],[DATETIME,DATETIME],[TIMESTAMP,TIMESTAMP],[INTERVAL,INTERVAL],[STRUCT,STRUCT],[ARRAY,ARRAY]}, but get [TIMESTAMP,INTEGER]\",\n    \"type\": \"ExpressionEvaluationException\"\n  },\n  \"status\": 400\n}

But, if you're using a JOIN, the behavior reverses, while the error becomes even more confusing than just "can't match type"
WHERE t.timestamp > timestamp('2001-05-07 00:00:00') now will not work and WHERE t.timestamp > 0 will with error

{\n  \"error\": {\n    \"reason\": \"Invalid SQL query\",\n    \"details\": \"Cannot invoke \\\"String.startsWith(String)\\\" because \\\"fieldName\\\" is null\",\n    \"type\": \"NullPointerException\"\n  },\n  \"status\": 400\n}
Related component

Search

To Reproduce
  1. Go to 'Dashboard -> Management -> Dev Tools'
  2. Paste the example below
  3. Run those queries one by one and observe the behavior
PUT /a
{
  "mappings": {
    "properties": {
      "some_id": {
        "type": "keyword"
      },
      "timestamp": {
        "type": "date"
      }
    }
  }
}

PUT /b
{
  "mappings": {
    "properties": {
      "some_id": {
        "type": "keyword"
      },
      "timestamp": {
        "type": "date"
      }
    }
  }
}

POST /a/_doc
{
  "some_id": "a1",
  "timestamp": "2024-12-15T10:00:00Z"
}

POST /b/_doc
{
  "some_id": "a1",
  "timestamp": "2024-12-15T11:00:00Z"
}

-- NO JOIN - TIMESTAMP works
POST /_plugins/_sql 
{
"query": "SELECT b.timestamp from b                                                WHERE b.timestamp > timestamp('2001-05-07 00:00:00');"
}
-- NO JOIN - INTEGER doesn't work
-- {\n  \"error\": {\n    \"reason\": \"Invalid SQL query\",\n    \"details\": \"> function expected {[BYTE,BYTE],[SHORT,SHORT],[INTEGER,INTEGER],[LONG,LONG],[FLOAT,FLOAT],[DOUBLE,DOUBLE],[STRING,STRING],[BOOLEAN,BOOLEAN],[DATE,DATE],[TIME,TIME],[DATETIME,DATETIME],[TIMESTAMP,TIMESTAMP],[INTERVAL,INTERVAL],[STRUCT,STRUCT],[ARRAY,ARRAY]}, but get [TIMESTAMP,INTEGER]\",\n    \"type\": \"ExpressionEvaluationException\"\n  },\n  \"status\": 400\n}
POST /_plugins/_sql 
{
"query": "SELECT b.timestamp from b                                                WHERE b.timestamp > 0;"
}

-- JOIN - INTEGER works
POST /_plugins/_sql 
{
"query": "SELECT b.timestamp from b JOIN a ON a.some_id = b.some_id WHERE b.timestamp > 0;"
}
-- JOIN - TIMESTAMP doesn't work
-- {\n  \"error\": {\n    \"reason\": \"Invalid SQL query\",\n    \"details\": \"Cannot invoke \\\"String.startsWith(String)\\\" because \\\"fieldName\\\" is null\",\n    \"type\": \"NullPointerException\"\n  },\n  \"status\": 400\n}
POST /_plugins/_sql 
{
"query": "SELECT b.timestamp from b JOIN a ON a.some_id = b.some_id WHERE b.timestamp > timestamp('2001-05-07 00:00:00');"
}
Expected behavior

Expected to see timestamp filtering work when using a join. Also, since it's a timestamp, i'd also expect INTEGER to work both ways

Additional Details

Plugins
AFAIK, sql, it's a default OpenSearch AWS installation

Screenshots
I've put the whole repro above with error results, so don't really need it

Host/Environment (please complete the following information):

  • Environment: AWS
  • Version: OpenSearch 2.17 (latest)
  • Service software version: OpenSearch_2_17_R20241112-P3 (latest)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the /_plugins/_sql endpoint and reproduce the four queries in the issue against the provided a and b mappings, both with and without the JOIN. Trace the timestamp comparison and type-conversion handling, then verify that timestamp filtering works consistently with joins and that invalid comparisons return a meaningful type error rather than a NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.