opensearch-project / opensearch-project/data-prepper
[BUG] Escape sequences aren't removed from regex
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
Escape characters aren't trimmed from regexPattern in the antlr grammar. Data prepper requires that the pattern ^\w*$ is escaped. It produces the following error message when the pattern is left unescaped.
/data-prepper/tree/main/data-prepper-plugins/http-source#authentication-configurations
line 1:9 token recognition error at: '^'
line 1:10 token recognition error at: '\'
line 1:11 token recognition error at: 'w*'
line 1:13 token recognition error at: '$"'
line 1:8 mismatched input '"' expecting {JsonPointer, EscapedJsonPointer, String}
When escaping \ and $, the parser no longer errors out, but the expression produced contains the escape sequences (^\\w*\$).
To Reproduce
Run data-prepper 2.11.0 with the following pipeline.yml
# pipeline.yml
main:
source:
http:
processor:
- add_entries:
entries:
- add_when: /msg =~ "^\\w*\$" # Here's the problematic expression
key: "matched"
value: "true"
sink:
- stdout:
Send it JSON with a msg field.
Expected behavior
The following msg value should satisfy the add_when expression using the regex operator:
{"msg":"word"}
Environment (please complete the following information):
- OS: [Debian Bookworm]
- Version [2.11]
Additional context
This value satisfies the regex used in the pipeline above.
{"msg":"\\wwww$"}
Adding a print statement to this lambda outputs the RegEx pattern with escape characters included.
https://github.com/opensearch-project/data-prepper/blob/52a1e6d912a8e1265b3ae212507e6edf6992a15f/data-prepper-expression/src/main/java/org/opensearch/dataprepper/expression/OperatorConfiguration.java#L23
I'm not super familiar with antrl, but from my understanding you'd want to remove escape characters within the listener. I'll spend a bit more time hacking away at the source, and submit a PR with whatever fix I find.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the provided pipeline.yml and inspect OperatorConfiguration.java around the linked lambda, then follow regex handling in ParseTreeEvaluatorListener.java. Confirm how the ANTLR parse tree represents escaped characters and verify that the add_when expression matches {"msg":"word"} without retaining unintended escape sequences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100