Graylog2 / Graylog2/graylog2-server
Handling OpenSearch mapping exceptions are unreasonably heavy
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
I had a typical incident where someone sent an attribute as long (JSON number), and afterwards started sending Strings. A symptom of this was logs flooded with errors like
```
OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [matches] of type [long] in document with id '752e2b45-fa90-11f0-b7fd-00505687ab33'. Preview of field's value: '314534|1']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=For input string: "314534|1"]];
```
The correct way to fix the root cause is to tell OpenSearch map the attribute in question to string/keyword. However, I noticed something undesirable.
Handling this exception is *insanely* CPU heavy for the Graylog process. A cluster that can ordinarily process 10k+ EPS will start collapsing at couple hundred EPS causing this Exception. The mongodb and opensearch stay healthy, Graylog uses all available cores.
## Expected Behavior
Graylog should probably cache the information the target index has a certain incompatible mapping, and bail out of the operation sooner. The time for this cache should be configurable. One extremely heavy operation once per minute sounds more reasonable.
## Current Behavior
Commit to OpenSearch. Exception gets bounced back. Exception gets handled, saved to Mongo. The logging operation to server log file is probably heavy, too.
## Possible Solution
Graylog should fail more gracefully, because now one misbehaving (sub-)system can impact the logging operations for everyone else. I'd add a caching type of feature and a status check for these operations.
A sysadmin is going to fix the root cause, and there'd be more time for them to work if the Graylog didn't crash under the load.
## Steps to Reproduce (for bugs)
1. Have a high EPS cluster.
2. Cause a bunch of mapping Exceptions.
## Context
I had to resolve an incident.
## Your Environment
* Graylog Version: 7.0.3
* OpenSearch Version: 2.19
* MongoDB Version: 7.x
* Operating System: RHEL
## Checklist
[no] This issue fix need to be backported.
[no] Does this issue have **security** implications?
Contributor guide
Assessment
This issue has not been assessed yet.