[RFC] - [Unified PPL] PPL Specification Versioning

1. Problem statements

• PPL will be used across multiple runtimes (OpenSearch, OpenSearch Serverless, Spark). Today, feature delivery and behavior changes can land at different times per engine, creating drift.
• Users need a stable contract for syntax, semantics, and optimizer-visible behavior across engine upgrades.
• We must ship features quickly without breaking existing queries.
• We need clear rules for when a change is “breaking,” how to deprecate safely, and how to migrate.

2. Proposed solution

2.1 Proposed solution

Single versioning surface: SemVer (x.y.z) for the PPL specification only.

• x (MAJOR): Breaking changes that can alter parse trees, result semantics, or observable default behaviors.
• y (MINOR): Backward-compatible features: new functions, syntax that cannot collide, opt-in optimizer features.
• z (PATCH): Bug fixes and clarifications that do not change behavior.

2.2 Show how the proposal solves the problems

A) Rapidly ship a new function: json_at(path, mode)

• Classification: Backward-compatible, additive ⇒ MINOR bump (e.g., 1.7.0 → 1.8.0).

• Spec delta in 1.8.0:

  • Signature: json_at(value, path, mode='strict') -> ANY
  • Semantics: JSONPath dialect, strict errors vs. lenient returns NULL.
  • Types: explicit return lattice; errors for invalid path/mode.

B) Change default NULL sort order (breaking)

• Classification: Observable behavior change ⇒ MAJOR bump (e.g., 1.11.0 → 2.0.0).

• Migration plan:

  1. Pre-work in MINOR (1.10.0):

     • Add explicit NULLS FIRST|LAST syntax (no default change).
     • ppl doctor warns on ORDER BY without explicit NULLs clause.

  2. Deprecation notice (1.11.0):

     • Document the upcoming default flip.

  3. MAJOR (2.0.0):

     • Flip default (e.g., ASC ⇒ NULLS LAST, DESC ⇒ NULLS FIRST).
     • Provide temporary runtime flag (e.g., ppl.legacy_nulls_order=true) for the 2.x line to ease rollbacks. Default is false.

  4. Next MAJOR (3.0.0):

     • Remove the legacy flag.

This RFC defines SemVer rules for the PPL specification, including breaking changes, additive features, patches, deprecation, and migration. Start by reviewing the proposal and its NULL ordering and json_at examples; done means the versioning policy and transition rules are agreed and actionable across the named runtimes.