opensearch-project / opensearch-project/sql

[FEATURE] Integrate SQL/PPL with query-insights plugin for query source tracking

Open
#5,677 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement feature
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?

The query-insights plugin captures all search queries as generic DSL queries. When SQL/PPL queries are executed through the _plugins/_sql or _plugins/_ppl endpoints, they generate one or more underlying DSL searches, but there is no way to trace these back to the original SQL/PPL statement. Operators cannot identify which SQL/PPL queries are consuming the most resources, causing latency spikes, or need optimization. The Top N Queries view shows only raw DSL bodies, requiring manual correlation.

What solution would you like?

Propagate SQL/PPL query metadata to the query-insights plugin via thread context headers so that:

  • Each DSL search generated by SQL/PPL is tagged with x-query-source (sql/ppl), x-original-query (the SQL/PPL text), and x-query-execution-id (a UUID correlating all DSL searches from one SQL/PPL execution)
  • SQL/PPL processing phase timings (parse, analyze, plan) are captured via x-query-phases header and exposed alongside the query record
  • Headers are registered as task headers via getTaskHeaders() so they propagate through the transport layer to the query-insights listener
  • The Calcite execution path (used by PPL) includes phase tracking alongside the legacy path (used by SQL)

What alternatives have you considered?

Pattern-matching the DSL source in the query-insights listener to detect SQL/PPL-generated queries — rejected as fragile, unreliable, and unable to recover the original SQL/PPL text.
Building a separate SQL-specific query tracking system — rejected as duplicative; the existing query-insights infrastructure already provides top-N ranking, storage, and dashboards.
Adding metadata at the REST layer only — rejected because it wouldn't propagate to child search tasks across nodes in a distributed cluster.

Do you have any additional context?

Implementation PR: https://github.com/opensearch-project/sql/pull/5636

Companion PRs in other repos:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the implementation PR in opensearch-project/sql#5636 and compare its work with the companion query-insights PRs #647 and #577. Review the _plugins/_sql and _plugins/_ppl paths and verify that generated searches propagate source, original-query, execution-id, and phase metadata through task headers, including Calcite and legacy execution paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.