elastic / elastic/support-diagnostics

[Feature Request] Add dedicated retention cap for archived index/search slowlogs

Open
#1,046 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
304
Forks
175
PR merge metrics
No merged PRs in 30d

Description

### Problem

The diagnostic currently collects the live `elasticsearch.log`, its archived
rollovers (`*.log.gz`, capped by `maxLogs`), and gc logs (capped by
`maxGcLogs`, see `CollectLogs.java`). Live slowlog files
(`_index_search_slowlog.json`, `_index_indexing_slowlog.json`)
are already picked up as part of the general log collection, but **archived/
rolled-over slowlog files have no dedicated retention cap** — they fall under
the same `maxLogs` bucket as unrelated `*.log.gz` archives (e.g. the main
elasticsearch log rollovers).

In practice, when investigating node instability caused by expensive search
queries (e.g. full data-stream scans, deep pagination, wide date ranges),
slowlogs are usually the fastest way to pinpoint the offending query — but
on a busy node where the main application log rotates frequently, older
slowlog archives can get evicted from the shared `maxLogs` cap before an
engineer gets a chance to look at them, even though they're often the more
useful artifact for this class of problem.

### Proposed Solution

Add a dedicated `maxSlowLogs` setting (default: `3`) in `diags.yml`'s
`log-settings`, mirroring the existing `maxGcLogs` pattern, and collect
search and indexing slowlog archives independently so one type can't crowd
out the other:

```yaml
log-settings:
maxLogs: 3
maxGcLogs: 3
maxSlowLogs: 3 # new
```

```yaml
# linuxOS / macOS logs:
slowlog-search: "ls -alt {{LOGPATH}} | grep '_index_search_slowlog' | awk '{print $9}'"
slowlog-index: "ls -alt {{LOGPATH}} | grep '_index_indexing_slowlog' | awk '{print $9}'"
```

```yaml
# winOS logs:
slowlog-search: "dir /l /b /o:-d /a:-d {{LOGPATH}}\\*_index_search_slowlog*"
slowlog-index: "dir /l /b /o:-d /a:-d {{LOGPATH}}\\*_index_indexing_slowlog*"
```

`CollectLogs.java` would collect these the same way it already does for
`gc`, using `DiagConfig.maxSlowLogs` for the entry limit.

Search and indexing slowlogs are intentionally collected via separate
commands (rather than one combined pattern) so that, e.g., a burst of
search slowlog rollovers can't push indexing slowlog archives out of the
result entirely — each type gets its own guaranteed slot.

### On sensitive content

Slowlogs can contain full query bodies, which may include sensitive field
values. This proposal does **not** introduce an opt-in flag — it follows
the same always-on approach already used for gc/log collection, since:

- Live slowlog files are already collected today with no flag, so this
doesn't change the existing sensitivity posture, only the retention
behavior of the archived files.
- Sanitization of slowlog content is a separate, already-tracked concern
(see "Related" below) and is best handled there rather than by gating
collection behind a new flag.

If maintainers would prefer this to be opt-in regardless, happy to add a
flag (e.g. `--includeSlowlogs`) instead — wanted to raise the question
here before deciding.

### Related

This is distinct from #410, which is about sanitizing/scrubbing slowlog
*content* after collection (e.g. removing query bodies via `scrub.yml`).
This proposal is only about giving *archived* slowlog files their own
retention cap, separate from the general `maxLogs` cap that `*.log.gz`
archives currently share — it doesn't change what content ends up in the
files themselves.

### Verification

Manually verified against a running 8.15.0 node: created 5 dummy search
slowlog archive files with identical timestamps, ran the diagnostic, and
confirmed only the 3 most recently modified were included in the output
archive, with indexing slowlog archives unaffected by the search slowlog
count.

I have a working patch for this (`diags.yml`, `DiagConfig.java`,
`CollectLogs.java`) and can open a PR once the approach here looks good —
wanted to check direction first per CONTRIBUTING.md.

Contributor guide

Open the contributing guide

Research direction

Start with the existing maxGcLogs handling in diags.yml, DiagConfig.java, and CollectLogs.java. Run the diagnostic with archived search and indexing slowlogs present; done means each type is collected independently up to maxSlowLogs, without changing the existing general-log or gc-log limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell
Domain
tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.