Graylog2 / Graylog2/graylog2-server

Run-away load when extractors fail casting on appliances

Open
#2,423 5 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

### Problem description

A series of really unfortunate events were experienced on a graylog instance today I was working on, which exposed an interest set of fail conditions.

Firstly, an extractor was defined, which had 2 components:
1. It extracted a timestamp from log4net file, which was randomly unable to be parsed
2. It had a field defined as {DATA:level;string}, which apparently the Graylog 2.0.1 OVA already has defined as integer from the default appliance input extractors, thus causing lots of exceptions due to being unable to cast (DEBUG|INFO|ERROR|etc) to integer

Normally I don't think this should have been a big issue, as I imagine extraction failures should be fine. But what then occurred is that because of all these extractor errors, the elasticsearch logs started being filled with exceptions, which were then picked up by the default appliance input.

Now, the default appliance input does not seem to handle multi-line exception messages, so due to each attempt at parsing a message input, it generated 10-20 log lines from the elasticsearch logs. This suddenly put the appliance into overtime, with process and output buffer being at 100%, with the elasticsearch log generating close 1000 lines per second.

This meant that the journal suddenly became really large quickly. The extractors were still failing, and generating more logs. To stop this, we tried to remove the extractors that were throwing exceptions, but the exceptions were still occuring, seemingly due to the extractors maybe being "cached" when parsing the journal? We also stopped the input itself.

We had to remove the actual input, and restart graylog-server, for the exceptions to stop occuring due to the extractors no longer being applying to the journal backlog. Once the journal was caught up, we could then resume normal operation.

In addition, in the scenario where datetimes are failing to parse correctly, the messages are silently dropped, and lots of exceptions messages are generated, and again logs become huge.
### Steps to reproduce the problem 1
1. Create an extractor which has a field defined as string, when a field already exists defined as integer
2. Start pushing messages at this input. At this point, you should get elasticsearch errors with casting/conversion errors
3. You should now see that you're getting a lot of exceptions being read from the default appliance input, which aren't being parsed as multi-line(Due to stacktrace), and thus generating a lot of messages
4. You should see no indication from the UI that anything is really wrong, other than buffers being full, and journal filling up
### Steps to reproduce the problem 2
1. Create an extractor which tries to parse out a string as datetime, but the format doesn't match what was used for the extractor
2. Start pushing messages at this input, with the bad timestamp type
3. You should now observe that exceptions are occurring, and messages are silently being dropped
- Graylog Version: 2.0.1 OVA

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.