elastic / elastic/support-diagnostics
[Feature Request] Add dedicated retention cap for archived index/search slowlogs
- Dominant language
- Java
- Stars
- 304
- Forks
- 175
- PR merge metrics
- No merged PRs in 30d
Description
### Problem
The diagnostic currently collects the live `elasticsearch.log`, its archived
rollovers (`*.log.gz`, capped by `maxLogs`), and gc logs (capped by
`maxGcLogs`, see `CollectLogs.java`). Live slowlog files
(`_index_search_slowlog.json`, `_index_indexing_slowlog.json`)
are already picked up as part of the general log collection, but **archived/
rolled-over slowlog files have no dedicated retention cap** — they fall under
the same `maxLogs` bucket as unrelated `*.log.gz` archives (e.g. the main
elasticsearch log rollovers).
In practice, when investigating node instability caused by expensive search
queries (e.g. full data-stream scans, deep pagination, wide date ranges),
slowlogs are usually the fastest way to pinpoint the offending query — but
on a busy node where the main application log rotates frequently, older
slowlog archives can get evicted from the shared `maxLogs` cap before an
engineer gets a chance to look at them, even though they're often the more
useful artifact for this class of problem.
### Proposed Solution
Add a dedicated `maxSlowLogs` setting (default: `3`) in `diags.yml`'s
`log-settings`, mirroring the existing `maxGcLogs` pattern, and collect
search and indexing slowlog archives independently so one type can't crowd
out the other:
```yaml
log-settings:
maxLogs: 3
maxGcLogs: 3
maxSlowLogs: 3 # new
```
```yaml
# linuxOS / macOS logs:
slowlog-search: "ls -alt {{LOGPATH}} | grep '_index_search_slowlog' | awk '{print $9}'"
slowlog-index: "ls -alt {{LOGPATH}} | grep '_index_indexing_slowlog' | awk '{print $9}'"
```
```yaml
# winOS logs:
slowlog-search: "dir /l /b /o:-d /a:-d {{LOGPATH}}\\*_index_search_slowlog*"
slowlog-index: "dir /l /b /o:-d /a:-d {{LOGPATH}}\\*_index_indexing_slowlog*"
```
`CollectLogs.java` would collect these the same way it already does for
`gc`, using `DiagConfig.maxSlowLogs` for the entry limit.
Search and indexing slowlogs are intentionally collected via separate
commands (rather than one combined pattern) so that, e.g., a burst of
search slowlog rollovers can't push indexing slowlog archives out of the
result entirely — each type gets its own guaranteed slot.
### On sensitive content
Slowlogs can contain full query bodies, which may include sensitive field
values. This proposal does **not** introduce an opt-in flag — it follows
the same always-on approach already used for gc/log collection, since:
- Live slowlog files are already collected today with no flag, so this
doesn't change the existing sensitivity posture, only the retention
behavior of the archived files.
- Sanitization of slowlog content is a separate, already-tracked concern
(see "Related" below) and is best handled there rather than by gating
collection behind a new flag.
If maintainers would prefer this to be opt-in regardless, happy to add a
flag (e.g. `--includeSlowlogs`) instead — wanted to raise the question
here before deciding.
### Related
This is distinct from #410, which is about sanitizing/scrubbing slowlog
*content* after collection (e.g. removing query bodies via `scrub.yml`).
This proposal is only about giving *archived* slowlog files their own
retention cap, separate from the general `maxLogs` cap that `*.log.gz`
archives currently share — it doesn't change what content ends up in the
files themselves.
### Verification
Manually verified against a running 8.15.0 node: created 5 dummy search
slowlog archive files with identical timestamps, ran the diagnostic, and
confirmed only the 3 most recently modified were included in the output
archive, with indexing slowlog archives unaffected by the search slowlog
count.
I have a working patch for this (`diags.yml`, `DiagConfig.java`,
`CollectLogs.java`) and can open a PR once the approach here looks good —
wanted to check direction first per CONTRIBUTING.md.
Contributor guide
Research direction
Start with the existing maxGcLogs handling in diags.yml, DiagConfig.java, and CollectLogs.java. Run the diagnostic with archived search and indexing slowlogs present; done means each type is collected independently up to maxSlowLogs, without changing the existing general-log or gc-log limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, shell
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100