opensearch-project / opensearch-project/sql
[FEATURE] Enable Calcite-only PPL commands (join, lookup, etc.) for Prometheus datasources
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Is your feature request related to a problem?
Non-OpenSearch datasources (e.g., Prometheus) cannot use any of the 31 Calcite-only PPL commands (join, lookup, flatten, expand, eventstats, streamstats, union, etc.). When Calcite is enabled, queries against Prometheus fall back to the V2 engine, which only supports source, where (limited), stats, and describe. This means users cannot correlate Prometheus metrics with OpenSearch data using join or lookup, severely limiting observability use cases.
What solution would you like?
Make PrometheusMetricTable participate in the Calcite query planning pipeline by implementing Calcite's TranslatableTable interface. This enables:
- All 31 Calcite-only PPL commands to work with Prometheus datasources
- Cross-datasource joins between OpenSearch indices and Prometheus metrics (federated queries)
- Filter pushdown of label equality conditions and time ranges into PromQL
- Dynamic sub-schema resolution for multi-part table references (
source = prometheus.metric_name)
What alternatives have you considered?
- Implementing join/lookup in the V2 engine — Would require refactoring the single-table assumption in
Planner.findTable(), creatingLogicalJoin/LogicalLookupnodes, and implementing physical join operators. Estimated 2-4 weeks, and wouldn't benefit from Calcite's built-in optimizer. - Using
ScannableTable(no pushdown) — Simpler but all filtering done in-memory by Calcite. Not viable for production since Prometheus queries need time-bounded PromQL. - Using
FilterableTable— Simpler filter pushdown model, butTranslatableTable.toRel()takes priority per Calcite's resolution order, and doesn't allow custom cost models.
Do you have any additional context?
- This was explicitly out of scope in the original Prometheus federation design (
docs/dev/datasource-prometheus.md: "Join Queries across datasources is out of scope") - The implementation leverages Calcite's native multi-schema federation — no changes to the join/lookup implementations themselves
- Label equality filters (
service = 'frontend') and time range comparisons on@timestampare pushed down to PromQL; all other conditions are evaluated in-memory by Calcite - Default time window is 1 hour (matching V2 behavior); full
TranslatableTablewith time range pushdown allows WHERE clause narrowing - Validated end-to-end with OTel demo:
source = logs | join prometheus.request ON ...works correctly
Yes..... this description is AI generated...but the initial request comes from real users...currently using elasticsearch ES|QL. The difference is, that the metrics are also stored in elasticsearch indices, while, in our current case, the metrics are stored in an associated prometheus compatible data source.
So...being able to do some lookup, join, etc. also on those metrics would be pretty nice
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/dev/datasource-prometheus.md and the existing PrometheusMetricTable, then trace how Calcite's TranslatableTable interface enters the query planning pipeline. Done means Prometheus supports the listed Calcite-only commands, federated joins, and the stated label and time-range pushdown behavior. Validate the result end-to-end with the OTel demo scenario described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, prometheus
- Domain
- backend-api-design, databases, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100