opensearch-project / opensearch-project/sql
[FEATURE] Make SQL align with PPL to support evaluate nested fields and update fields.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Is your feature request related to a problem?
Currently in OpenSearch, SQL doesn't support update or evaluate nested fields like PPL. To make them align, we may need to make SQL support these features as well.
e.g.
- Update fields
PPL supports updating the fields while SQL produces a new field with the same field_name which may leads to error.
# PPL
POST _plugins/_ppl/_explain
{
"query": """
source=opensearch_dashboards_sample_data_flights | eval FlightTimeMin = 1
"""
}
# response
{
"root": {
"name": "ProjectOperator",
"description": {
"fields": "[FlightNum, Origin, OriginLocation, DestLocation, FlightDelay, DistanceMiles, FlightTimeMin, OriginWeather, dayOfWeek, AvgTicketPrice, Carrier, FlightDelayMin, OriginRegion, DestAirportID, FlightDelayType, timestamp, Dest, FlightTimeHour, Cancelled, DistanceKilometers, OriginCityName, DestWeather, OriginCountry, DestCountry, DestRegion, DestCityName, OriginAirportID]"
},
"children": [
{
"name": "EvalOperator",
"description": {
"expressions": {
"FlightTimeMin": "1"
}
},
"children": [
{
"name": "OpenSearchIndexScan",
"description": {
"request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":10000,"timeout":"1m"}, searchDone=false)"""
},
"children": []
}
]
}
]
}
}
# SQL
POST _plugins/_sql
{
"query": "SELECT *, 1 as FlightTimeMin FROM opensearch_dashboards_sample_data_flights LIMIT 5"
}
# response error
"{\n \"error\": {\n \"reason\": \"Invalid SQL query\",\n \"details\": \"Multiple entries with same key: FlightTimeMin=1 and FlightTimeMin=1030.7704\",\n \"type\": \"IllegalArgumentException\"\n },\n \"status\": 400\n}"
- Evaluate nested fields
PPL supports evaluating fields from left to right while SQL cannot resolve the reference.
# PPL
POST _plugins/_ppl/_explain
{
"query": """
source=opensearch_dashboards_sample_data_flights | eval FlightTimeMinPlusOne = FlightTimeMin + 1, FlightTimeMinPlusTwo = FlightTimeMinPlusOne + 1 | fields FlightTimeMinPlusOne, FlightTimeMinPlusTwo
"""
}
# response
{
"root": {
"name": "ProjectOperator",
"description": {
"fields": "[FlightTimeMinPlusOne, FlightTimeMinPlusTwo]"
},
"children": [
{
"name": "EvalOperator",
"description": {
"expressions": {
"FlightTimeMinPlusTwo": "+(FlightTimeMinPlusOne, 1)",
"FlightTimeMinPlusOne": "+(FlightTimeMin, 1)"
}
},
"children": [
{
"name": "OpenSearchIndexScan",
"description": {
"request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":10000,"timeout":"1m"}, searchDone=false)"""
},
"children": []
}
]
}
]
}
}
# SQL
POST _plugins/_sql/_explain
{
"query": "SELECT (FlightTimeMin + 1) as FlightTimeMinPlusOne, (FlightTimeMinPlusOne + 1) as FlightTimeMinPlusTwo FROM opensearch_dashboards_sample_data_flights LIMIT 5"
}
# response error
"{\n \"error\": {\n \"reason\": \"Invalid SQL query\",\n \"details\": \"can't resolve Symbol(namespace=FIELD_NAME, name=FlightTimeMinPlusOne) in type env\",\n \"type\": \"SemanticCheckException\"\n },\n \"status\": 400\n}"
What solution would you like?
These 2 features are all supported in EvalOperator. But when compiling SQL, it won't use EvalOperator now but only ProjectOperator, which contributes to the differences above.
To make them align, there could be 2 optional solutions:
- Make SQL generates EvalOperator as well like PPL
- Enhance ProjectOperator to support these 2 features.
What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.
Do you have any additional context?
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing SQL compilation from the _plugins/_sql and _plugins/_sql/_explain endpoints, comparing the ProjectOperator path with the EvalOperator behavior shown in the examples. Determine whether SQL should generate EvalOperator or enhance ProjectOperator, then verify that updates to existing fields and left-to-right nested evaluations produce the expected results without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100