opensearch-project / opensearch-project/sql

Audit field availability for new keywords added with the 3.3 improvements

Open
#4,280 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

maintenance testing v3.3.0
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

The Situation

This issue came up with #4185, #4113, #4120, and probably others:

In the PPL and SQL grammars, there's a keywordsCanBeId entry:

https://github.com/opensearch-project/sql/blob/ab02d56a9cbabfe8ec95957dec64d481438ce043/ppl/src/main/antlr/OpenSearchPPLParser.g4#L1314

We've been adding a lot of new keywords to the Lexer, but haven't been adding them all to this entry. This means that you can't use those keywords as fields anymore without escaping. You get an error similar to this:

[field] is not a valid term at this part of the query: '...output=output inner.field' <-- HERE. Expecting tokens: ID

In the above case, the field keyword was added in #4109 and caused test failures for #4185 after merge, because field was used as a generic field name.

The Fix

We should audit the new keywords added as part of the 3.3 rush, and verify that all appropriate new keywords are also available as field names.

We could also consider finding a way to refactor such that we don't need to manually update this allowlist to begin with.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the keywordsCanBeId entry in ppl/src/main/antlr/OpenSearchPPLParser.g4 and compare it with the new keywords added to the PPL and SQL lexer grammars during the 3.3 improvements. Review the failures described in #4185 and related issues, then verify that appropriate keywords remain usable as field names without escaping. Done means the audit is complete and the affected parsing tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
compilers, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.