opensearch-project / opensearch-project/data-prepper

[BUG] Escape sequences aren't removed from regex

Open
#5,652 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug

Escape characters aren't trimmed from regexPattern in the antlr grammar. Data prepper requires that the pattern ^\w*$ is escaped. It produces the following error message when the pattern is left unescaped.

/data-prepper/tree/main/data-prepper-plugins/http-source#authentication-configurations
line 1:9 token recognition error at: '^'
line 1:10 token recognition error at: '\'
line 1:11 token recognition error at: 'w*'
line 1:13 token recognition error at: '$"'
line 1:8 mismatched input '"' expecting {JsonPointer, EscapedJsonPointer, String}

When escaping \ and $, the parser no longer errors out, but the expression produced contains the escape sequences (^\\w*\$).

To Reproduce
Run data-prepper 2.11.0 with the following pipeline.yml

# pipeline.yml
main:
  source:
    http:
  processor:
    - add_entries:
        entries:
          - add_when: /msg =~ "^\\w*\$" # Here's the problematic expression
            key: "matched"
            value: "true"
  sink:
    - stdout:

Send it JSON with a msg field.

Expected behavior

The following msg value should satisfy the add_when expression using the regex operator:

{"msg":"word"}

Environment (please complete the following information):

  • OS: [Debian Bookworm]
  • Version [2.11]

Additional context

This value satisfies the regex used in the pipeline above.

{"msg":"\\wwww$"}

Adding a print statement to this lambda outputs the RegEx pattern with escape characters included.
https://github.com/opensearch-project/data-prepper/blob/52a1e6d912a8e1265b3ae212507e6edf6992a15f/data-prepper-expression/src/main/java/org/opensearch/dataprepper/expression/OperatorConfiguration.java#L23

I'm not super familiar with antrl, but from my understanding you'd want to remove escape characters within the listener. I'll spend a bit more time hacking away at the source, and submit a PR with whatever fix I find.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the provided pipeline.yml and inspect OperatorConfiguration.java around the linked lambda, then follow regex handling in ParseTreeEvaluatorListener.java. Confirm how the ANTLR parse tree represents escaped characters and verify that the add_when expression matches {"msg":"word"} without retaining unintended escape sequences.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.