Graylog2 / Graylog2/graylog2-server

Handling OpenSearch mapping exceptions are unreasonably heavy

Open
#24,831 7 comments 0 reactions 0 assignees View on GitHub
7.3 candidate bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

I had a typical incident where someone sent an attribute as long (JSON number), and afterwards started sending Strings. A symptom of this was logs flooded with errors like

```
OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [matches] of type [long] in document with id '752e2b45-fa90-11f0-b7fd-00505687ab33'. Preview of field's value: '314534|1']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=For input string: "314534|1"]];
```

The correct way to fix the root cause is to tell OpenSearch map the attribute in question to string/keyword. However, I noticed something undesirable.

Handling this exception is *insanely* CPU heavy for the Graylog process. A cluster that can ordinarily process 10k+ EPS will start collapsing at couple hundred EPS causing this Exception. The mongodb and opensearch stay healthy, Graylog uses all available cores.

## Expected Behavior

Graylog should probably cache the information the target index has a certain incompatible mapping, and bail out of the operation sooner. The time for this cache should be configurable. One extremely heavy operation once per minute sounds more reasonable.

## Current Behavior

Commit to OpenSearch. Exception gets bounced back. Exception gets handled, saved to Mongo. The logging operation to server log file is probably heavy, too.

## Possible Solution

Graylog should fail more gracefully, because now one misbehaving (sub-)system can impact the logging operations for everyone else. I'd add a caching type of feature and a status check for these operations.

A sysadmin is going to fix the root cause, and there'd be more time for them to work if the Graylog didn't crash under the load.

## Steps to Reproduce (for bugs)

1. Have a high EPS cluster.
2. Cause a bunch of mapping Exceptions.

## Context

I had to resolve an incident.

## Your Environment

* Graylog Version: 7.0.3
* OpenSearch Version: 2.19
* MongoDB Version: 7.x
* Operating System: RHEL

## Checklist
[no] This issue fix need to be backported.
[no] Does this issue have **security** implications?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.