opensearch-project / opensearch-project/data-prepper
DeleteEntry Processor support for parsing json object
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
The DeleteEntry processor uses GenericExpressionEvaluator and it doesn't support deleting key with DeleteEntry processor when the expression path refers to json object.
log-pipeline:
source:
http:
path: "/test/path"
processor:
- delete_entries:
with_keys: [ "responseElements" ]
delete_when: '/responseElements == null'
- date:
from_time_received: true
destination: "@timestamp"
sink:
- stdout:
curl -X POST http://localhost:2021/test/path -H 'Content-Type: application/json; charset=utf-8' -d '[{"log":"sample log", "responseElements": {"a":"a", "b": "b" }}]'
2023-08-07T11:10:58,081 [log-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - Encountered exception during pipeline log-pipeline processing
org.opensearch.dataprepper.expression.ExpressionEvaluationException: Unable to evaluate statement "/responseElements == null"
at org.opensearch.dataprepper.expression.GenericExpressionEvaluator.evaluate(GenericExpressionEvaluator.java:41) ~[data-prepper-expression-2.3.2.jar:?]
at org.opensearch.dataprepper.expression.ExpressionEvaluator.evaluateConditional(ExpressionEvaluator.java:28) ~[data-prepper-api-2.3.2.jar:?]
at org.opensearch.dataprepper.plugins.processor.mutateevent.DeleteEntryProcessor.doExecute(DeleteEntryProcessor.java:40) ~[mutate-event-processors-2.3.2.jar:?]
at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.3.2.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) ~[micrometer-core-1.10.5.jar:1.10.5]
at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) ~[data-prepper-api-2.3.2.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:115) ~[data-prepper-core-2.3.2.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:50) [data-prepper-core-2.3.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Describe the solution you'd like
Add support for parsing Json object in LiteralTypeConversionsConfiguration that is used by GenericExpressionEvaluator.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with LiteralTypeConversionsConfiguration.java and trace how GenericExpressionEvaluator evaluates the /responseElements == null expression for DeleteEntryProcessor. Reproduce the JSON-object example from the issue, add support for parsing that value, and verify that the DeleteEntry processor handles the expression without the reported evaluation exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- Half a day
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100