opensearch-project / opensearch-project/sql
[BUG] Parsed field data missing when source field not included
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
What is the bug?
When using the PPL parse command, if the source field that was parsed is not included in the subsequent fields command, the parsed field data is missing from the final result.
This behavior appears to be by design in the code: ProjectOperator.java#L76.
How can one reproduce the bug?
POST parse_command_test/_doc
{
"@timestamp": "2025-01-04T04:00:00Z",
"line": {
"@message": "Request from AWS for client ID 123 is OVER_LIMIT"
}
}
POST _plugins/_ppl
{
"query": """
search source=parse_command_test
| parse line.@message 'Request from (?<service>.+) for client ID (?<clientId>.+) is OVER_LIMIT'
| fields @timestamp, service, clientId
"""
}
{
"schema": [
{
"name": "@timestamp",
"type": "timestamp"
},
{
"name": "service",
"type": "string"
},
{
"name": "clientId",
"type": "string"
}
],
"datarows": [
[
"2025-01-04 04:00:00"
]
],
"total": 1,
"size": 1
}
What is the expected behavior?
- If intentional: Provide clear documentation to explain why parsed field data depends on the inclusion of the original field in the fields command.
- Otherwise: Modify the behavior to ensure parsed fields remain available even if the original field is excluded.
Do you have any screenshots?
N/A
Do you have any additional context?
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with core/src/main/java/org/opensearch/sql/planner/physical/ProjectOperator.java around line 76 and reproduce the PPL query from the issue against parse_command_test. Trace how the fields command handles parsed fields when line.@message is omitted, then determine whether the intended result is a behavior change or documentation update. Done means the chosen behavior is implemented and the reproduction no longer loses service and clientId, or the dependency is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100