Graylog2 / Graylog2/graylog2-server
Graylog doesn't accept a newline as GELF short_message
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Graylog is not accepting a newline "\n" as a valid GELF short_message, throwing an error message that is not very helpful.
## Expected Behavior
I would prefer graylog to accept the newline, or to throw a warning instead of an error.
Many programs (I had the issue with mongodb) log empty lines and it's hard for general log collectors to parse and filter all the possible whitespace and non-useful strings. I understand the error if the field is not set, but maybe empty strings, whitespace and the like could be a warning or info instead of an error if you don't want to accept them.
What I would really like to see is a more descriptive error message and maybe better documentation on what is not allowed in the short_message field.
## Current Behavior
Trying to ingest a gelf json with `"short_message":"\n"` throws an error
The only clue for debugging is the message `has empty mandatory "short_message" field`:
```
2018-06-12 14:40:45,775 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=972f15f0-6e4e-11e8-9b74-a28a23afbb85, journalOffset=17468033, codec=gelf, payloadSize=107, timestamp=2018-06-12T14:40:45.775Z, remoteAddress=/127.0.0.1:45498} on input <5b1f239f2e4b68000198bd9c>.
2018-06-12 14:40:45,775 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=972f15f0-6e4e-11e8-9b74-a28a23afbb85, journalOffset=17468033, codec=gelf, payloadSize=107, timestamp=2018-06-12T14:40:45.775Z, remoteAddress=/127.0.0.1:45498}
java.lang.IllegalArgumentException: GELF message <972f15f0-6e4e-11e8-9b74-a28a23afbb85> (received from <127.0.0.1:45498>) has empty mandatory "short_message" field.
at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
I was collecting messages from a very busy fluentd instance, and it has been hard to find what kind of messages were throwing the errors, since I was looking blindly for empty strings.
## Possible Solution
I have 3 suggestions:
* Accept "\n" as valid short_message or lower its log level.
* Print the full message in the error stack trace, at least in DEBUG or TRACE log levels. I think this will be most useful. It's very hard to know what messages are the problematic ones when you have a big volume of them
* Could we have a more detailed description of valid short_messages? This is what the documentation says now (http://docs.graylog.org/en/2.4/pages/gelf.html#gelf-payload-specification):
```
short_message string (UTF-8)
a short descriptive message; MUST be set by client library.
```
## Context
I am collecting logs from a kubernetes cluster using fluentd daemonset. I am sending the logs to graylog using https://github.com/bodhi-space/fluent-plugin-gelf-hs, which checks for the short_message being not empty, but is not parsing whitespace. I think now the plugin is doing right, since the docs doesn't specify what to check for.
## Your Environment
* Graylog Version: 2.4.5
* Elasticsearch Version: 5.6.2
* MongoDB Version: 3.4.6
* Operating System:
* Browser version:
Thank you very much!
Contributor guide
Assessment
This issue has not been assessed yet.