opensearch-project / opensearch-project/sql
[FEATURE] Enrichment Explain Extended Results
Open
Nobody has claimed this yet.
enhancement
PPL
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Is your feature request related to a problem?
First, read issue
There is currently no effective way to determine which field is actually being used when debugging this issue.
- The EXPLAIN results are not human-readable and require better formatting and refinement.
- The EXPLAIN output is missing key debugging information. For example, it may show a function like
REX_EXTRACT($0, status=(?<statusCode>\\d+), 1), but it does not reveal how $0 is fetched from OpenSearch, which is essential for diagnosing the problem.
###
POST {{baseUrl}}/_plugins/_ppl
Content-Type: application/x-ndjson
{
"query": "explain extended source=log* | rex field=msg 'status=(?<statusCode>\\d+)' | where statusCode=200"
}
{
"calcite": {
"logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n LogicalProject(msg=[$0], index=[$1], statusCode=[$8])\n LogicalFilter(condition=[=(SAFE_CAST($8), 200.0E0)])\n LogicalProject(msg=[$0], index=[$1], _id=[$2], _index=[$3], _score=[$4], _maxscore=[$5], _sort=[$6], _routing=[$7], statusCode=[REX_EXTRACT($0, 'status=(?<statusCode>\\d+)', 1)])\n CalciteLogicalIndexScan(table=[[OpenSearch, log*]])\n",
"physical": "EnumerableCalc(expr#0..1=[{inputs}], expr#2=['status=(?<statusCode>\\d+)'], expr#3=[1], expr#4=[REX_EXTRACT($t0, $t2, $t3)], proj#0..1=[{exprs}], $f2=[$t4])\n CalciteEnumerableIndexScan(table=[[OpenSearch, log*]], PushDownContext=[[PROJECT->[msg, index], SCRIPT->=(SAFE_CAST(REX_EXTRACT($0, 'status=(?<statusCode>\\d+)', 1)), 200.0E0), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"script\":{\"script\":{\"source\":\"{\\\"langType\\\":\\\"calcite\\\",\\\"script\\\":\\\"{\\\\n \\\\\\\"op\\\\\\\": {\\\\n \\\\\\\"name\\\\\\\": \\\\\\\"=\\\\\\\",\\\\n \\\\\\\"kind\\\\\\\": \\\\\\\"EQUALS\\\\\\\",\\\\n \\\\\\\"syntax\\\\\\\": \\\\\\\"BINARY\\\\\\\"\\\\n },\\\\n \\\\\\\"operands\\\\\\\": [\\\\n {\\\\n \\\\\\\"op\\\\\\\": {\\\\n \\\\\\\"name\\\\\\\": \\\\\\\"SAFE_CAST\\\\\\\",\\\\n \\\\\\\"kind\\\\\\\": \\\\\\\"SAFE_CAST\\\\\\\",\\\\n \\\\\\\"syntax\\\\\\\": \\\\\\\"SPECIAL\\\\\\\"\\\\n },\\\\n \\\\\\\"operands\\\\\\\": [\\\\n {\\\\n \\\\\\\"op\\\\\\\": {\\\\n \\\\\\\"name\\\\\\\": \\\\\\\"REX_EXTRACT\\\\\\\",\\\\n \\\\\\\"kind\\\\\\\": \\\\\\\"OTHER_FUNCTION\\\\\\\",\\\\n \\\\\\\"syntax\\\\\\\": \\\\\\\"FUNCTION\\\\\\\"\\\\n },\\\\n \\\\\\\"operands\\\\\\\": [\\\\n {\\\\n \\\\\\\"input\\\\\\\": 0,\\\\n \\\\\\\"name\\\\\\\": \\\\\\\"$0\\\\\\\"\\\\n },\\\\n {\\\\n \\\\\\\"literal\\\\\\\": \\\\\\\"status=(?<statusCode>\\\\\\\\\\\\\\\\d+)\\\\\\\",\\\\n \\\\\\\"type\\\\\\\": {\\\\n \\\\\\\"type\\\\\\\": \\\\\\\"CHAR\\\\\\\",\\\\n \\\\\\\"nullable\\\\\\\": false,\\\\n \\\\\\\"precision\\\\\\\": 25\\\\n }\\\\n },\\\\n {\\\\n \\\\\\\"literal\\\\\\\": 1,\\\\n \\\\\\\"type\\\\\\\": {\\\\n \\\\\\\"type\\\\\\\": \\\\\\\"INTEGER\\\\\\\",\\\\n \\\\\\\"nullable\\\\\\\": false\\\\n }\\\\n }\\\\n ],\\\\n \\\\\\\"class\\\\\\\": \\\\\\\"org.opensearch.sql.expression.function.UserDefinedFunctionBuilder$1\\\\\\\",\\\\n \\\\\\\"type\\\\\\\": {\\\\n \\\\\\\"type\\\\\\\": \\\\\\\"VARCHAR\\\\\\\",\\\\n \\\\\\\"nullable\\\\\\\": true,\\\\n \\\\\\\"precision\\\\\\\": 2000\\\\n },\\\\n \\\\\\\"deterministic\\\\\\\": true,\\\\n \\\\\\\"dynamic\\\\\\\": false\\\\n }\\\\n ],\\\\n \\\\\\\"type\\\\\\\": {\\\\n \\\\\\\"type\\\\\\\": \\\\\\\"DOUBLE\\\\\\\",\\\\n \\\\\\\"nullable\\\\\\\": true\\\\n }\\\\n },\\\\n {\\\\n \\\\\\\"literal\\\\\\\": 200.0,\\\\n \\\\\\\"type\\\\\\\": {\\\\n \\\\\\\"type\\\\\\\": \\\\\\\"DOUBLE\\\\\\\",\\\\n \\\\\\\"nullable\\\\\\\": false\\\\n }\\\\n }\\\\n ]\\\\n}\\\"}\",\"lang\":\"opensearch_compounded_script\",\"params\":{\"utcTimestamp\":1761319006844113000}},\"boost\":1.0}},\"_source\":{\"includes\":[\"msg\",\"index\"],\"excludes\":[]}}, requestedTotalSize=10000, pageSize=null, startFrom=0)])\n",
"extended": "public org.apache.calcite.linq4j.Enumerable bind(final org.apache.calcite.DataContext root) {\n final org.opensearch.sql.opensearch.storage.scan.CalciteEnumerableIndexScan v1stashed = (org.opensearch.sql.opensearch.storage.scan.CalciteEnumerableIndexScan) root.get(\"v1stashed\");\n final org.apache.calcite.linq4j.Enumerable _inputEnumerable = v1stashed.scan();\n return new org.apache.calcite.linq4j.AbstractEnumerable(){\n public org.apache.calcite.linq4j.Enumerator enumerator() {\n return new org.apache.calcite.linq4j.Enumerator(){\n public final org.apache.calcite.linq4j.Enumerator inputEnumerator = _inputEnumerable.enumerator();\n public void reset() {\n inputEnumerator.reset();\n }\n\n public boolean moveNext() {\n return inputEnumerator.moveNext();\n }\n\n public void close() {\n inputEnumerator.close();\n }\n\n public Object current() {\n final Object[] current = (Object[]) inputEnumerator.current();\n final String input_value1 = current[0] == null ? null : current[0].toString();\n return new Object[] {\n current[0],\n current[1],\n input_value1 == null ? null : org.opensearch.sql.expression.function.udf.RexExtractFunction.extractGroup(input_value1, \"status=(?<statusCode>\\\\d+)\", 1)};\n }\n\n };\n }\n\n };\n}\n\n\npublic Class getElementType() {\n return java.lang.Object[].class;\n}\n\n\n"
}
}
What solution would you like?
opensearch_compounded_script results should
- readable.
- has type info.
e.g.SAFE_CAST(REX_EXTRACT(docValue(msg), status=(?<statusCode>\\d+), 1), DOUBLE)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue 4659, then inspect the EXPLAIN EXTENDED output shown here, especially the opensearch_compounded_script result. Done means the result is readable, includes type information, and represents the source field as docValue(msg) in the requested form.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100