elastic / elastic/elasticsearch
[Ingest Processors][Painless]Better error handling on if conditions in Ingest Processors
- Dominant language
- Java
- Stars
- 77.9k
- Forks
- 26.1k
- PR merge metrics
- PR metrics pending
Description
When making beat modules with large ingest pipelines, its often hard to find a single typo in if conditions since the error message you get when referencing a non-existing context object does not point you in the right direction.
Example:
```
- set:
if: "ctx._temp_.cisco.message_id == '304001'"
field: "event.outcome"
value: allow
```
Compared to:
```
- set:
if: "ctx._temp_.cisco?.message_id == '304001'"
field: "event.outcome"
value: allow
```
Imagine when you have hundreds of lines with conditions and the only error you often have to go on is something like:
```
{"@timestamp":"2020-06-07T14:36:33.626Z", "log.level":"DEBUG", "message":"unexpected exception during publication", "service.name":"ES_ECS","process.thread.name":"elasticsearch[b4435c79b09e][management][T#1]","log.logger":"org.elasticsearch.action.support.master.TransportMasterNodeAction","type":"server","cluster.uuid":"wraGqOByQ1OgkiLg3gPHLw","node.id":"nozzmTD8Qzuj8g21lTrTrQ","node.name":"b4435c79b09e","cluster.name":"docker-cluster","error.type":"java.lang.NullPointerException","error.message":"Cannot invoke \"Object.getClass()\" because \"config\" is null","error.stack_trace":"java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"config\" is null\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:405)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:74)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:319)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:225)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:84)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:91)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:158)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:151)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:232)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:209)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:187)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:179)\n\tat org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:571)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1098)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1176)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1156)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:228)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:286)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:737)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:691)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"config\" is null\n\t\t... 30 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"config\" is null\n\t\t... 30 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n"}
```
The example and the error message above is not directly related. The error message itself is when you have a malformed config, but the output is the same. It will complain trying to get `Object.getClass()` of something that does not exist or is null.
`Object.getClass()` which runs for both config mapping and pipeline conditions could at least tell what it is trying to access and where, it would save anyone making pipelines a big amount of time.
Contributor guide
Research direction
Start by reading ConfigurationUtils.readProcessor and readProcessorConfigs, then trace how Pipeline.create and IngestService validation surface malformed processor and pipeline-condition errors. The issue names these entry points and expects errors to identify the accessed context or configuration path instead of reporting only a null Object.getClass() failure; no test file is specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100