opensearch-project / opensearch-project/sql

[FEATURE]Move PPL Language Spec Outside SQL As an independed Repository

Open
#2,821 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?

Description

Piped Processing Language (PPL), enables users with exploration and discovery of their data, and finding search patterns in data stored in multiple locations (S3, OpenSearch, Prometheus) using a set of commands delimited by pipes (|).

During the past year the SQL/PPL team was focusing on the following tasks:

  • Transforming PPL to become OpenSearch default query language (specifically for logs/traces/metrics signals)
  • Promoting PPL as a viable candidate for the proposed CNCF Observability universal query language.
  • Seamlessly Interact with different datasources such as S3 / Prometheus / data-lake leveraging spark execution.
  • Using spark's federative capabilities as a general purpose query engine to facilitate complex queries including joins
  • Improve and promote PPL to become extensible and general purpose query language to be adopted by the community

For historical reasons, PPL language specifications is currently located in the OpenSearch SQL repository.
In addition the PPL specifications code & documents are not present as an independed (jar) artifact but is bundled with the SQL plugin as an OpenSearch Zip file.

History of PPL

PPL has become a general purpose pipeline language that finds attraction and usage in many places in the log analytics echo-system.
Its originated as a language that has a dedicated OpenSearch driver that was the only execution engine that could run the language inside opensearch.

Since PPL has evolved and is now able to run on top of Spark as a fully qualified query language.

What solution would you like?

ppl logical architecture

This PR has the goal of decoupling the PPL language specifications and documentation away from the OpenSearch SQL Plugin and move it (back) into the PPL dedicated repository.

This repository should contain the following:

  • ANTLR specifications
  • Documentations
  • Planned changes and general language issues

The repository should release a jar artifact which is not coupled with the OpenSearch release cadence and should have its own versioning.

A major advantage for this approach would be to allow different execution engines (drivers) such as OpenSearch, Spark, Prometheus and more to be decoupled from the SQL repository and maintain a independed trail of support for PPL commands and features.


Do you have any additional context?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review docs/user/ppl/index.rst, the linked PPL repository, and issues 1222 and 30 to understand the existing specification and migration context. Define the repository contents and release boundary, including ANTLR specifications, documentation, planned language changes, and an independently versioned jar. Done means PPL is separated from the SQL plugin without coupling its release cadence.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
build-system, documentation, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.