opensearch-project / opensearch-project/sql
[BUG] SQL confusing type conversion for TIMESTAMP when using join
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Describe the bug
So given 2 indexes, with a matching field (to be used in a JOIN) and a timestamp field. When querying without a join, a WHERE clause like WHERE t.timestamp > timestamp('2001-05-07 00:00:00') will work and WHERE t.timestamp > 0 won't work since it's a TIMESTAMP field, that's what opensearch tell you
{\n \"error\": {\n \"reason\": \"Invalid SQL query\",\n \"details\": \"> function expected {[BYTE,BYTE],[SHORT,SHORT],[INTEGER,INTEGER],[LONG,LONG],[FLOAT,FLOAT],[DOUBLE,DOUBLE],[STRING,STRING],[BOOLEAN,BOOLEAN],[DATE,DATE],[TIME,TIME],[DATETIME,DATETIME],[TIMESTAMP,TIMESTAMP],[INTERVAL,INTERVAL],[STRUCT,STRUCT],[ARRAY,ARRAY]}, but get [TIMESTAMP,INTEGER]\",\n \"type\": \"ExpressionEvaluationException\"\n },\n \"status\": 400\n}
But, if you're using a JOIN, the behavior reverses, while the error becomes even more confusing than just "can't match type"
WHERE t.timestamp > timestamp('2001-05-07 00:00:00') now will not work and WHERE t.timestamp > 0 will with error
{\n \"error\": {\n \"reason\": \"Invalid SQL query\",\n \"details\": \"Cannot invoke \\\"String.startsWith(String)\\\" because \\\"fieldName\\\" is null\",\n \"type\": \"NullPointerException\"\n },\n \"status\": 400\n}
Related component
Search
To Reproduce
- Go to 'Dashboard -> Management -> Dev Tools'
- Paste the example below
- Run those queries one by one and observe the behavior
PUT /a
{
"mappings": {
"properties": {
"some_id": {
"type": "keyword"
},
"timestamp": {
"type": "date"
}
}
}
}
PUT /b
{
"mappings": {
"properties": {
"some_id": {
"type": "keyword"
},
"timestamp": {
"type": "date"
}
}
}
}
POST /a/_doc
{
"some_id": "a1",
"timestamp": "2024-12-15T10:00:00Z"
}
POST /b/_doc
{
"some_id": "a1",
"timestamp": "2024-12-15T11:00:00Z"
}
-- NO JOIN - TIMESTAMP works
POST /_plugins/_sql
{
"query": "SELECT b.timestamp from b WHERE b.timestamp > timestamp('2001-05-07 00:00:00');"
}
-- NO JOIN - INTEGER doesn't work
-- {\n \"error\": {\n \"reason\": \"Invalid SQL query\",\n \"details\": \"> function expected {[BYTE,BYTE],[SHORT,SHORT],[INTEGER,INTEGER],[LONG,LONG],[FLOAT,FLOAT],[DOUBLE,DOUBLE],[STRING,STRING],[BOOLEAN,BOOLEAN],[DATE,DATE],[TIME,TIME],[DATETIME,DATETIME],[TIMESTAMP,TIMESTAMP],[INTERVAL,INTERVAL],[STRUCT,STRUCT],[ARRAY,ARRAY]}, but get [TIMESTAMP,INTEGER]\",\n \"type\": \"ExpressionEvaluationException\"\n },\n \"status\": 400\n}
POST /_plugins/_sql
{
"query": "SELECT b.timestamp from b WHERE b.timestamp > 0;"
}
-- JOIN - INTEGER works
POST /_plugins/_sql
{
"query": "SELECT b.timestamp from b JOIN a ON a.some_id = b.some_id WHERE b.timestamp > 0;"
}
-- JOIN - TIMESTAMP doesn't work
-- {\n \"error\": {\n \"reason\": \"Invalid SQL query\",\n \"details\": \"Cannot invoke \\\"String.startsWith(String)\\\" because \\\"fieldName\\\" is null\",\n \"type\": \"NullPointerException\"\n },\n \"status\": 400\n}
POST /_plugins/_sql
{
"query": "SELECT b.timestamp from b JOIN a ON a.some_id = b.some_id WHERE b.timestamp > timestamp('2001-05-07 00:00:00');"
}
Expected behavior
Expected to see timestamp filtering work when using a join. Also, since it's a timestamp, i'd also expect INTEGER to work both ways
Additional Details
Plugins
AFAIK, sql, it's a default OpenSearch AWS installation
Screenshots
I've put the whole repro above with error results, so don't really need it
Host/Environment (please complete the following information):
- Environment: AWS
- Version: OpenSearch 2.17 (latest)
- Service software version: OpenSearch_2_17_R20241112-P3 (latest)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the /_plugins/_sql endpoint and reproduce the four queries in the issue against the provided a and b mappings, both with and without the JOIN. Trace the timestamp comparison and type-conversion handling, then verify that timestamp filtering works consistently with joins and that invalid comparisons return a meaningful type error rather than a NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100