Introduce a standard mechanism to avoid logging multiple times the same line.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
When an error in managing an event that doesn't impact the execution is encountered , it could happen that many log lines with exact same text is printed cluttering the log itself with meaningless content and shading other more important lines (like in #19066).
An example is an error in evaluating the size of batch at (version `9.2.8`):
https://github.com/elastic/logstash/blob/3e2bf2d3b86563d5ba1e733478b4cca42d9d1612/logstash-core/src/main/java/org/logstash/execution/QueueReadClientBatchMetrics.java#L72-L79
It repeatedly log the line
```
Failed to calculate batch byte size for metrics
```
for each batch.
Stack trace sample
```
Failed to calculate batch byte size for metrics
java.lang.IllegalArgumentException: Unsupported type encountered in estimateMemory: org.logstash.Timestamp. Please ensure all objects passed to estimateMemory are of supported types. Refer to the ConvertedMap.estimateMemory method for the list of supported types.
at org.logstash.ConvertedMap.estimateMemory(ConvertedMap.java:275) ~[logstash-core.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.IdentityHashMap$ValueSpliterator.forEachRemaining(IdentityHashMap.java:1565) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.LongPipeline.reduce(LongPipeline.java:498) ~[?:?]
at java.util.stream.LongPipeline.sum(LongPipeline.java:456) ~[?:?]
at org.logstash.ConvertedMap.estimateMemory(ConvertedMap.java:177) ~[logstash-core.jar:?]
at org.logstash.ConvertedMap.estimateMemory(ConvertedMap.java:224) ~[logstash-core.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.IdentityHashMap$ValueSpliterator.forEachRemaining(IdentityHashMap.java:1565) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.LongPipeline.reduce(LongPipeline.java:498) ~[?:?]
at java.util.stream.LongPipeline.sum(LongPipeline.java:456) ~[?:?]
at org.logstash.ConvertedMap.estimateMemory(ConvertedMap.java:177) ~[logstash-core.jar:?]
at org.logstash.Event.estimateMemory(Event.java:579) ~[logstash-core.jar:?]
at org.logstash.execution.QueueReadClientBatchMetrics.updateBatchSizeMetric(QueueReadClientBatchMetrics.java:72) ~[logstash-core.jar:?]
at org.logstash.execution.QueueReadClientBatchMetrics.updateBatchMetrics(QueueReadClientBatchMetrics.java:63) ~[logstash-core.jar:?]
at org.logstash.execution.QueueReadClientBase.startMetrics(QueueReadClientBase.java:211) ~[logstash-core.jar:?]
at org.logstash.ext.JrubyAckedReadClientExt.readBatch(JrubyAckedReadClientExt.java:89) ~[logstash-core.jar:?]
at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:82) ~[logstash-core.jar:?]
```
### Feature description
Provide a functionality to print such lines or one at warn level or if debug is enabled it can be printed each time the error is encountered.
#### Implementation details
There could be an accumulator of log lines, a sort cache of lines already logged. If that's the implementatio also a sort of LRU should be used to avoid excessive memory consumption, or maybe consider a Bloom filter to test for already logged lines.
Contributor guide
Assessment
This issue has not been assessed yet.