opensearch-project / opensearch-project/sql
[FEATURE] Integrate SQL/PPL with query-insights plugin for query source tracking
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Is your feature request related to a problem?
The query-insights plugin captures all search queries as generic DSL queries. When SQL/PPL queries are executed through the _plugins/_sql or _plugins/_ppl endpoints, they generate one or more underlying DSL searches, but there is no way to trace these back to the original SQL/PPL statement. Operators cannot identify which SQL/PPL queries are consuming the most resources, causing latency spikes, or need optimization. The Top N Queries view shows only raw DSL bodies, requiring manual correlation.
What solution would you like?
Propagate SQL/PPL query metadata to the query-insights plugin via thread context headers so that:
- Each DSL search generated by SQL/PPL is tagged with x-query-source (sql/ppl), x-original-query (the SQL/PPL text), and x-query-execution-id (a UUID correlating all DSL searches from one SQL/PPL execution)
- SQL/PPL processing phase timings (parse, analyze, plan) are captured via x-query-phases header and exposed alongside the query record
- Headers are registered as task headers via getTaskHeaders() so they propagate through the transport layer to the query-insights listener
- The Calcite execution path (used by PPL) includes phase tracking alongside the legacy path (used by SQL)
What alternatives have you considered?
Pattern-matching the DSL source in the query-insights listener to detect SQL/PPL-generated queries — rejected as fragile, unreliable, and unable to recover the original SQL/PPL text.
Building a separate SQL-specific query tracking system — rejected as duplicative; the existing query-insights infrastructure already provides top-N ranking, storage, and dashboards.
Adding metadata at the REST layer only — rejected because it wouldn't propagate to child search tasks across nodes in a distributed cluster.
Do you have any additional context?
Implementation PR: https://github.com/opensearch-project/sql/pull/5636
Companion PRs in other repos:
- Query Insights plugin (aggregation, storage): https://github.com/opensearch-project/query-insights/pull/647
- Query Insights Dashboards (UI): https://github.com/opensearch-project/query-insights-dashboards/pull/577
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the implementation PR in opensearch-project/sql#5636 and compare its work with the companion query-insights PRs #647 and #577. Review the _plugins/_sql and _plugins/_ppl paths and verify that generated searches propagate source, original-query, execution-id, and phase metadata through task headers, including Calcite and legacy execution paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, data, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100