elastic / elastic/logstash

Discuss Jstack dumps and how we can make interpreting them easier.

Open
#5,281 3 comments 0 reactions 0 assignees View on GitHub
discuss
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

[Read this](http://blog.takipi.com/supercharged-jstack-how-to-debug-your-servers-at-100mph/)

TL;DR
We do rename the thread. The article suggests running a watchdog thread that would run dumps from within LS and logs the captured output. The dump is triggered by a performance degradation.

We could do this and improve on it by taking one dump when performance is nominal and several in succession when its not. We could log to a different file (if log file arg ti set) by adding a `jstack` suffix to the file name or maybe its better to use the cabin LS logger.

We should also use `LOGSTASH_CORE_VERSION` in the name of the watchdog thread.

Also we should build a knowledge base in google docs of as many LS jstack dumps as we can and put them in versioned folders.

What else can we do?

With the PQ, each job (batch) and event will probably have a unique identifier. Should we at least INFO log the job id as it enters the worker thread? Should each filter/output plugin INFO log each event id as it processes it. <- This maybe useful when the jstack dumps are mingled with LS logging.

Contributor guide

Open the contributing guide

Research direction

Start with the linked jstack article and the issue's proposals for watchdog-triggered dumps, logging, thread naming, and job or event identifiers. No repository files or tests are named; done would require an agreed, scoped design for collecting and interpreting Logstash jstack dumps rather than an open-ended discussion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.