opensearch-project / opensearch-project/sql

[FEATURE] Enrichment Explain Extended Results

Open
#4,660 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?
First, read issue

There is currently no effective way to determine which field is actually being used when debugging this issue.

  • The EXPLAIN results are not human-readable and require better formatting and refinement.
  • The EXPLAIN output is missing key debugging information. For example, it may show a function like REX_EXTRACT($0, status=(?<statusCode>\\d+), 1), but it does not reveal how $0 is fetched from OpenSearch, which is essential for diagnosing the problem.
###
POST {{baseUrl}}/_plugins/_ppl
Content-Type: application/x-ndjson

{
  "query": "explain extended source=log* | rex field=msg 'status=(?<statusCode>\\d+)' | where statusCode=200"
}

{
  "calcite": {
    "logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n  LogicalProject(msg=[$0], index=[$1], statusCode=[$8])\n    LogicalFilter(condition=[=(SAFE_CAST($8), 200.0E0)])\n      LogicalProject(msg=[$0], index=[$1], _id=[$2], _index=[$3], _score=[$4], _maxscore=[$5], _sort=[$6], _routing=[$7], statusCode=[REX_EXTRACT($0, 'status=(?<statusCode>\\d+)', 1)])\n        CalciteLogicalIndexScan(table=[[OpenSearch, log*]])\n",
    "physical": "EnumerableCalc(expr#0..1=[{inputs}], expr#2=['status=(?<statusCode>\\d+)'], expr#3=[1], expr#4=[REX_EXTRACT($t0, $t2, $t3)], proj#0..1=[{exprs}], $f2=[$t4])\n  CalciteEnumerableIndexScan(table=[[OpenSearch, log*]], PushDownContext=[[PROJECT->[msg, index], SCRIPT->=(SAFE_CAST(REX_EXTRACT($0, 'status=(?<statusCode>\\d+)', 1)), 200.0E0), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"script\":{\"script\":{\"source\":\"{\\\"langType\\\":\\\"calcite\\\",\\\"script\\\":\\\"{\\\\n  \\\\\\\"op\\\\\\\": {\\\\n    \\\\\\\"name\\\\\\\": \\\\\\\"=\\\\\\\",\\\\n    \\\\\\\"kind\\\\\\\": \\\\\\\"EQUALS\\\\\\\",\\\\n    \\\\\\\"syntax\\\\\\\": \\\\\\\"BINARY\\\\\\\"\\\\n  },\\\\n  \\\\\\\"operands\\\\\\\": [\\\\n    {\\\\n      \\\\\\\"op\\\\\\\": {\\\\n        \\\\\\\"name\\\\\\\": \\\\\\\"SAFE_CAST\\\\\\\",\\\\n        \\\\\\\"kind\\\\\\\": \\\\\\\"SAFE_CAST\\\\\\\",\\\\n        \\\\\\\"syntax\\\\\\\": \\\\\\\"SPECIAL\\\\\\\"\\\\n      },\\\\n      \\\\\\\"operands\\\\\\\": [\\\\n        {\\\\n          \\\\\\\"op\\\\\\\": {\\\\n            \\\\\\\"name\\\\\\\": \\\\\\\"REX_EXTRACT\\\\\\\",\\\\n            \\\\\\\"kind\\\\\\\": \\\\\\\"OTHER_FUNCTION\\\\\\\",\\\\n            \\\\\\\"syntax\\\\\\\": \\\\\\\"FUNCTION\\\\\\\"\\\\n          },\\\\n          \\\\\\\"operands\\\\\\\": [\\\\n            {\\\\n              \\\\\\\"input\\\\\\\": 0,\\\\n              \\\\\\\"name\\\\\\\": \\\\\\\"$0\\\\\\\"\\\\n            },\\\\n            {\\\\n              \\\\\\\"literal\\\\\\\": \\\\\\\"status=(?<statusCode>\\\\\\\\\\\\\\\\d+)\\\\\\\",\\\\n              \\\\\\\"type\\\\\\\": {\\\\n                \\\\\\\"type\\\\\\\": \\\\\\\"CHAR\\\\\\\",\\\\n                \\\\\\\"nullable\\\\\\\": false,\\\\n                \\\\\\\"precision\\\\\\\": 25\\\\n              }\\\\n            },\\\\n            {\\\\n              \\\\\\\"literal\\\\\\\": 1,\\\\n              \\\\\\\"type\\\\\\\": {\\\\n                \\\\\\\"type\\\\\\\": \\\\\\\"INTEGER\\\\\\\",\\\\n                \\\\\\\"nullable\\\\\\\": false\\\\n              }\\\\n            }\\\\n          ],\\\\n          \\\\\\\"class\\\\\\\": \\\\\\\"org.opensearch.sql.expression.function.UserDefinedFunctionBuilder$1\\\\\\\",\\\\n          \\\\\\\"type\\\\\\\": {\\\\n            \\\\\\\"type\\\\\\\": \\\\\\\"VARCHAR\\\\\\\",\\\\n            \\\\\\\"nullable\\\\\\\": true,\\\\n            \\\\\\\"precision\\\\\\\": 2000\\\\n          },\\\\n          \\\\\\\"deterministic\\\\\\\": true,\\\\n          \\\\\\\"dynamic\\\\\\\": false\\\\n        }\\\\n      ],\\\\n      \\\\\\\"type\\\\\\\": {\\\\n        \\\\\\\"type\\\\\\\": \\\\\\\"DOUBLE\\\\\\\",\\\\n        \\\\\\\"nullable\\\\\\\": true\\\\n      }\\\\n    },\\\\n    {\\\\n      \\\\\\\"literal\\\\\\\": 200.0,\\\\n      \\\\\\\"type\\\\\\\": {\\\\n        \\\\\\\"type\\\\\\\": \\\\\\\"DOUBLE\\\\\\\",\\\\n        \\\\\\\"nullable\\\\\\\": false\\\\n      }\\\\n    }\\\\n  ]\\\\n}\\\"}\",\"lang\":\"opensearch_compounded_script\",\"params\":{\"utcTimestamp\":1761319006844113000}},\"boost\":1.0}},\"_source\":{\"includes\":[\"msg\",\"index\"],\"excludes\":[]}}, requestedTotalSize=10000, pageSize=null, startFrom=0)])\n",
    "extended": "public org.apache.calcite.linq4j.Enumerable bind(final org.apache.calcite.DataContext root) {\n  final org.opensearch.sql.opensearch.storage.scan.CalciteEnumerableIndexScan v1stashed = (org.opensearch.sql.opensearch.storage.scan.CalciteEnumerableIndexScan) root.get(\"v1stashed\");\n  final org.apache.calcite.linq4j.Enumerable _inputEnumerable = v1stashed.scan();\n  return new org.apache.calcite.linq4j.AbstractEnumerable(){\n      public org.apache.calcite.linq4j.Enumerator enumerator() {\n        return new org.apache.calcite.linq4j.Enumerator(){\n            public final org.apache.calcite.linq4j.Enumerator inputEnumerator = _inputEnumerable.enumerator();\n            public void reset() {\n              inputEnumerator.reset();\n            }\n\n            public boolean moveNext() {\n              return inputEnumerator.moveNext();\n            }\n\n            public void close() {\n              inputEnumerator.close();\n            }\n\n            public Object current() {\n              final Object[] current = (Object[]) inputEnumerator.current();\n              final String input_value1 = current[0] == null ? null : current[0].toString();\n              return new Object[] {\n                  current[0],\n                  current[1],\n                  input_value1 == null ? null : org.opensearch.sql.expression.function.udf.RexExtractFunction.extractGroup(input_value1, \"status=(?<statusCode>\\\\d+)\", 1)};\n            }\n\n          };\n      }\n\n    };\n}\n\n\npublic Class getElementType() {\n  return java.lang.Object[].class;\n}\n\n\n"
  }
}

What solution would you like?
opensearch_compounded_script results should

  1. readable.
  2. has type info.
    e.g. SAFE_CAST(REX_EXTRACT(docValue(msg), status=(?<statusCode>\\d+), 1), DOUBLE)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading issue 4659, then inspect the EXPLAIN EXTENDED output shown here, especially the opensearch_compounded_script result. Done means the result is readable, includes type information, and represents the source field as docValue(msg) in the requested form.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.