opensearch-project / opensearch-project/data-prepper
[BUG] Regex matching with Data Prepper Expression throws error when using $
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
Trying to use a regex in a Data Prepper expression with a $ in the Regex is causing this error:
2023-10-17T16:24:04,064 [simple-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.pipeline.router.RouteEventEvaluator - Failed to evaluate route. This route will not be applied to any events.
org.opensearch.dataprepper.expression.ExpressionEvaluationException: Unable to evaluate statement "/app =~ "-service$""
at org.opensearch.dataprepper.expression.GenericExpressionEvaluator.evaluate(GenericExpressionEvaluator.java:41) ~[data-prepper-expression-2.5.0.jar:?]
at org.opensearch.dataprepper.expression.ExpressionEvaluator.evaluateConditional(ExpressionEvaluator.java:28) ~[data-prepper-api-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.router.RouteEventEvaluator.findMatchedRoutes(RouteEventEvaluator.java:64) [data-prepper-core-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.router.RouteEventEvaluator.evaluateEventRoutes(RouteEventEvaluator.java:45) [data-prepper-core-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.router.Router.route(Router.java:39) [data-prepper-core-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.Pipeline.publishToSinks(Pipeline.java:335) [data-prepper-core-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.postToSink(ProcessWorker.java:151) [data-prepper-core-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:133) [data-prepper-core-2.5.0.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:60) [data-prepper-core-2.5.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.opensearch.dataprepper.expression.ParseTreeCompositeException
at org.opensearch.dataprepper.expression.ParseTreeParser.createParseTree(ParseTreeParser.java:78) ~[data-prepper-expression-2.5.0.jar:?]
at org.opensearch.dataprepper.expression.ParseTreeParser.parse(ParseTreeParser.java:101) ~[data-prepper-expression-2.5.0.jar:?]
at org.opensearch.dataprepper.expression.ParseTreeParser.parse(ParseTreeParser.java:27) ~[data-prepper-expression-2.5.0.jar:?]
at org.opensearch.dataprepper.expression.MultiThreadParser.parse(MultiThreadParser.java:35) ~[data-prepper-expression-2.5.0.jar:?]
at org.opensearch.dataprepper.expression.MultiThreadParser.parse(MultiThreadParser.java:20) ~[data-prepper-expression-2.5.0.jar:?]
at org.opensearch.dataprepper.expression.GenericExpressionEvaluator.evaluate(GenericExpressionEvaluator.java:37) ~[data-prepper-expression-2.5.0.jar:?]
... 13 more
Caused by: org.opensearch.dataprepper.expression.ExceptionOverview: Multiple exceptions (2)
|-- org.antlr.v4.runtime.InputMismatchException: null
at org.antlr.v4.runtime.DefaultErrorStrategy.sync(DefaultErrorStrategy.java:270)
|-- org.antlr.v4.runtime.LexerNoViableAltException: null
at org.antlr.v4.runtime.atn.LexerATNSimulator.failOrAccept(LexerATNSimulator.java:309)
In this case the condition was:
app-logs: "/app =~ \"-service$\""
This is similar to the example in the documentation:
https://github.com/opensearch-project/data-prepper/blob/main/docs/expression_syntax.md#reference-table
To Reproduce
Steps to reproduce the behavior:
Create a pipeline with a configuration similar to the following:
simple-pipeline:
workers: 2
delay: "5000"
source:
http:
path: "/ingest"
route:
- app-logs: "/app =~ \"-service$\""
sink:
- stdout:
routes:
- app-logs
Send data to the pipeline
curl -k -XPOST -H "Content-Type: application/json" -d '[{"app": "-service"}]' http://localhost:2021/ingest
Check logs
Expected behavior
The provided expression should be able to be parsed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the expression-evaluation path shown in the trace: GenericExpressionEvaluator, MultiThreadParser, and ParseTreeParser. Reproduce the route with the documented pipeline configuration and the "-service$" regex, then verify that the expression parses successfully and the route matches the supplied event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100