opensearch-project / opensearch-project/sql

[Enhancement] Support Illegal Character in Regex Name Group

Open
#4,549 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Description

Currently, regex based extraction commands which are using java regex library has a limitation of including special characters such as (-, _ ,@) in the named captured group for creating a new column in the result site. Here are some related issues:

PR: https://github.com/opensearch-project/sql/pull/4434 enhanced the experience of unify the error handling of this for both parse and rex commands. Here is the current behavior

curl -X POST "localhost:9200/_plugins/_ppl" -H 'Content-Type: application/json' -d'{
    "query": "source=accounts | rex field=email \"(?<username>[^@]+)@(?<domain_name>[^.]+)\" | fields email, username, domain_name | head 3"
  }' | jq

{
  "error": {
    "reason": "Invalid Query",
    "details": "Invalid capture group name 'domain_name'. Java regex group names must start with a letter and contain only letters and digits.",
    "type": "IllegalArgumentException"
  },
  "status": 400
}

However, Coming from the https://github.com/opensearch-project/sql/pull/4434#issuecomment-3399182076 @ykmr1224 pointed we should be able to support the invalid characters by rewriting regex and map extracted values back to original name.

Expected Behavior

e.g.: (?<user_name>.+)(?<username>.+)(?<username1>.+) => (?<username2>.+)(?<username>(?<username1>.+), mapping = {username2 => user_name, username => username, username1 => username1}

Exit Criteria
  • Proper testing cover all the edge cases of re-writing - reference to https://github.com/opensearch-project/sql/pull/4434#issuecomment-3395550765
  • Double check the debugging flows (e.g. /_explain and server log) make sure this will not be lead into any confusions
  • Performance testing to make sure no notable performance downgrade
  • Update the documentations if the behavior changed (both parse and rex)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading PR 4434 and the linked issues 3944 and 4467 to understand the existing parse and rex error handling. Trace the regex rewriting and name-mapping behavior, then verify edge-case tests, _explain output, server logs, performance, and updated documentation for both commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.