Graylog2 / Graylog2/graylog2-server
Debugging broken extractors / inputs
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
As of now, if something goes wrong it is very hard to debug.
Example: I see this in log:
```
2021-02-19 13:19:11,403 ERROR: org.graylog2.indexer.messages.Messages - Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2021-02-19 13:19:11,403 WARN : org.graylog2.indexer.messages.Messages - Failed to index message: index= id=<0d95cbe0-72b5-11eb-9ecc-0242ac100008> error=<{"type":"mapper_parsing_exception","reason":"failed to parse field [level] of type [long]","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"WARN\""}}>
2021-02-19 13:19:11,404 ERROR: org.graylog2.outputs.BlockingBatchedESOutput - Unable to flush message buffer
java.lang.ClassCastException: Cannot cast java.lang.String to org.joda.time.DateTime
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_265]
at org.graylog2.plugin.Message.getFieldAs(Message.java:643) ~[graylog.jar:?]
at org.graylog2.plugin.Message.getTimestamp(Message.java:349) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.propagateFailure(Messages.java:365) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:192) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:159) ~[graylog.jar:?]
at org.graylog2.outputs.ElasticSearchOutput.writeMessageEntries(ElasticSearchOutput.java:111) ~[graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:129) [graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.forceFlushIfTimedout(BlockingBatchedESOutput.java:154) [graylog.jar:?]
at org.graylog2.periodical.BatchedElasticSearchOutputFlushThread.doRun(BatchedElasticSearchOutputFlushThread.java:82) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_265]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
```
## Expected Behavior
There is some information about the message source, message content, etc.
## Current Behavior
- We cannot estimate, which extractor / which message leads to this errors (nor if it is for only one message or many)
- Calling graylog to get the message by ID we get a 404 (apparently if the message that has not been processed is not in the database?)
## Possible Solution
Say in logs:
- Which Field is concerned
- What is the whole message
## Context
We have 5 inputs with ~10 extractors each and now there are errors in the logs and we cannot estimate which application produces it
## Your Environment
* Graylog Version:3.3.8
* Java Version:
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:
Contributor guide
Assessment
This issue has not been assessed yet.