Graylog2 / Graylog2/graylog2-server

Debugging broken extractors / inputs

Open
#10,109 1 comment 0 reactions 1 assignee Claimed by @jalogisch View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

As of now, if something goes wrong it is very hard to debug.

Example: I see this in log:

```
2021-02-19 13:19:11,403 ERROR: org.graylog2.indexer.messages.Messages - Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2021-02-19 13:19:11,403 WARN : org.graylog2.indexer.messages.Messages - Failed to index message: index= id=<0d95cbe0-72b5-11eb-9ecc-0242ac100008> error=<{"type":"mapper_parsing_exception","reason":"failed to parse field [level] of type [long]","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"WARN\""}}>
2021-02-19 13:19:11,404 ERROR: org.graylog2.outputs.BlockingBatchedESOutput - Unable to flush message buffer
java.lang.ClassCastException: Cannot cast java.lang.String to org.joda.time.DateTime
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_265]
at org.graylog2.plugin.Message.getFieldAs(Message.java:643) ~[graylog.jar:?]
at org.graylog2.plugin.Message.getTimestamp(Message.java:349) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.propagateFailure(Messages.java:365) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:192) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:159) ~[graylog.jar:?]
at org.graylog2.outputs.ElasticSearchOutput.writeMessageEntries(ElasticSearchOutput.java:111) ~[graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:129) [graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.forceFlushIfTimedout(BlockingBatchedESOutput.java:154) [graylog.jar:?]
at org.graylog2.periodical.BatchedElasticSearchOutputFlushThread.doRun(BatchedElasticSearchOutputFlushThread.java:82) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_265]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
```

## Expected Behavior
There is some information about the message source, message content, etc.

## Current Behavior

- We cannot estimate, which extractor / which message leads to this errors (nor if it is for only one message or many)
- Calling graylog to get the message by ID we get a 404 (apparently if the message that has not been processed is not in the database?)

## Possible Solution
Say in logs:
- Which Field is concerned
- What is the whole message

## Context
We have 5 inputs with ~10 extractors each and now there are errors in the logs and we cannot estimate which application produces it

## Your Environment

* Graylog Version:3.3.8
* Java Version:
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.