elastic / elastic/elasticsearch

[Ingest Processors][Painless]Better error handling on if conditions in Ingest Processors

Open
#57,793 4 comments 0 reactions 0 assignees View on GitHub
:Distributed/Ingest Node >enhancement
Dominant language
Java
Stars
77.9k
Forks
26.1k
PR merge metrics
PR metrics pending

Description

When making beat modules with large ingest pipelines, its often hard to find a single typo in if conditions since the error message you get when referencing a non-existing context object does not point you in the right direction.

Example:

```
- set:
if: "ctx._temp_.cisco.message_id == '304001'"
field: "event.outcome"
value: allow
```

Compared to:

```
- set:
if: "ctx._temp_.cisco?.message_id == '304001'"
field: "event.outcome"
value: allow
```

Imagine when you have hundreds of lines with conditions and the only error you often have to go on is something like:

```
{"@timestamp":"2020-06-07T14:36:33.626Z", "log.level":"DEBUG", "message":"unexpected exception during publication", "service.name":"ES_ECS","process.thread.name":"elasticsearch[b4435c79b09e][management][T#1]","log.logger":"org.elasticsearch.action.support.master.TransportMasterNodeAction","type":"server","cluster.uuid":"wraGqOByQ1OgkiLg3gPHLw","node.id":"nozzmTD8Qzuj8g21lTrTrQ","node.name":"b4435c79b09e","cluster.name":"docker-cluster","error.type":"java.lang.NullPointerException","error.message":"Cannot invoke \"Object.getClass()\" because \"config\" is null","error.stack_trace":"java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"config\" is null\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:405)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:74)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:319)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:225)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:84)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:91)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:158)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:151)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:232)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:209)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:187)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:179)\n\tat org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:571)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1098)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1176)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1156)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:228)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:286)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:737)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:691)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"config\" is null\n\t\t... 30 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"config\" is null\n\t\t... 30 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n\tSuppressed: org.elasticsearch.ElasticsearchParseException: property isn't a map, but of type [java.lang.String]\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:315)\n\t\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:404)\n\t\t... 29 more\n"}
```

The example and the error message above is not directly related. The error message itself is when you have a malformed config, but the output is the same. It will complain trying to get `Object.getClass()` of something that does not exist or is null.

`Object.getClass()` which runs for both config mapping and pipeline conditions could at least tell what it is trying to access and where, it would save anyone making pipelines a big amount of time.

Contributor guide

Open the contributing guide

Research direction

Start by reading ConfigurationUtils.readProcessor and readProcessorConfigs, then trace how Pipeline.create and IngestService validation surface malformed processor and pipeline-condition errors. The issue names these entry points and expects errors to identify the accessed context or configuration path instead of reporting only a null Object.getClass() failure; no test file is specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.