opensearch-project / opensearch-project/sql

[FEATURE] PPL: Gracefully handle non-existent fields in queries instead of throwing errors

Open
#4,197 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

calcite enhancement PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?
PPL queries fail when attempting to reference non-existent fields, particularly problematic when working with OpenTelemetry (OTEL) data where not all standard fields may be instrumented.

What solution would you like?
Modify the PPL query engine to gracefully handle non-existent fields by ignoring them instead of throwing errors. This would allow queries to:

  • Execute successfully even when referencing fields that don't exist in the index
  • Return results based on the fields that do exist
  • Skip or ignore clauses referencing non-existent fields

What alternatives have you considered?
Current workaround requires additional preprocessing steps:

  • Fetching available fields using index pattern API:
const indexPattern = await data.indexPatterns.get(indexPatternId);
const fieldNames = indexPattern.fields;
  • Filtering query clauses to only include existing fields This creates unnecessary overhead in query generation and makes the code more complex.

Do you have any additional context?

  • This is particularly important for OpenTelemetry use cases where the standard defines many possible fields, but applications may only instrument a subset
  • Similar functionality exists in other query languages (e.g., Elasticsearch's exists query)
  • Would improve user experience by removing the need to know exact schema before querying
  • Current behavior forces users to write defensive code and add complexity to their applications

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the PPL query engine entry point and tracing how references to non-existent fields currently produce errors. Define how missing fields should be handled across filters and other clauses, then verify that queries return results from existing fields without throwing. No specific files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.