logzio / logzio/logzio-java-sender

Logzio appender stops sending log events

Open
#45 25 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
15
Forks
4
Avg merge
12m
Merged PRs (30d)
3

Description

Everything goes well with shipping the log evens to logz.io, until after a day of three the appender stops sending events to logzio.
This has happened on multiple occasions.

We enabled debug logging of the logzio appender and sender. A snippet of the output in the log file is captured below.

Looking at the log file, it tries to drain the queue with one logging event in it, but never sends it. After that it does not attempt to drain the queue again.
And the log line that should be send is identical to other log lines it already sent earlier in the run of the application, so we have no reason to suspect that it has anything to do with the content of that log line.

The version we use:
` compile "io.logz.logback:logzio-logback-appender:1.0.22"`

```
20:31:12,222 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:12,547 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 1658
20:31:17,548 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:22,548 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:27,548 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:32,548 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:37,548 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
^[[2m2019-06-29 20:31:40.158^[[0;39m ^[[32mDEBUG^[[0;39m ^[[35m8^[[0;39m ^[[2m---^[[0;39m ^[[2m[ scheduler-3]^[[0;39m ^[[36m [APP LOG LINE]
20:31:42,548 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:42,950 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 1658
20:31:47,950 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
20:31:52,951 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
^[[2m2019-06-29 20:31:53.562^[[0;39m ^[[32mDEBUG^[[0;39m ^[[35m8^[[0;39m ^[[2m---^[[0;39m ^[[2m[ scheduler-17]^[[0;39m ^[[36m [APP LOG LINE]
20:31:57,951 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - DEBUG: Attempting to drain queue
.. more log lines
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with LogzioLogbackAppender and the sender behavior described in the debug output, using version 1.0.22 as the reproduction target. Investigate why a queued event can remain after a drain attempt and reproduce the behavior over time. Done means queued log events continue to be sent reliably rather than leaving the appender inactive.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.