Analyze Logstash RSS memory with `glibc` and `jemalloc`

# Logstash RSS grows over time on Linux/glibc while JVM heap stays bounded — native allocator retention?

TL;DR:
- if you are observing high RSS memory and Logstash is running on Linux/glib, use `-XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=60000` JVM options which periodically trims the native memory back to OS. `UnlockExperimentalVMOptions` option isn't required if JDK is 21.0.3+.
- if you want to know where the memory arena allocations are happening, use `jemalloc`, set `dirty_decay_ms:-1` (telling do not auto trim native memory to see where retains pages are left) profile to figure out. Memory allocation depends on pipeline definition (which plugins are used)

## Summary
On Logstash **8.19.13** running on Linux (glibc), process **RSS (RES) grows steadily over time and is never released back to the OS**, eventually reaching many GB, even though the JVM heap is bounded (`-Xms1g -Xmx1g`, pipeline buffers on heap). Forcing a native trim (`System.trim_native_heap`) reclaims a large amount of RES, which points at **native allocator retention rather than a JVM-heap leak**. We're trying to confirm the mechanism and the right mitigation, and to rule out a true native leak.

## Environment
- Logstash 8.19.13 (bundled JDK 21), `pipeline.buffer.type: heap`
- Linux x86_64, glibc (default allocator), no `MALLOC_ARENA_MAX` / `LD_PRELOAD` set, used default what glibc 2.39 version has.

## Symptoms
- `top`/`ps` RES climbs monotonically and does not come back down at idle.
- JVM heap usage itself remains within `-Xmx`; no OOME from the Java heap.

## What we have found so far
- Reproduced the retention on a **Linux/glibc** container: RES ratchets up and holds even after large GCs (heap freed, RES not returned).
- The Logstash **Node API / Puma** threads are short-lived (created per scrape, reaped); thread count is bounded — not a thread leak, though the create/destroy churn does allocate native thread stacks via glibc.
- Allocation profiling (async-profiler `nativemem`) at idle/light load is dominated by **JIT compilation** (transient, code-cache bounded); workload paths (AWS SDK, sockets) are tiny.
- `System.trim_native_heap` or JVM `UnlockExperimentalVMOptions` drops RES substantially (e.g., ~11.7 GB → ~4.7 GB), with VIRT/SHR roughly unchanged.
- jemalloc profiling (`jeprof` diff of dumps over time) with `dirty_decay_ms:-1` (-1 means no native trim claim to make sure jeprof files are generated overtime): warmup window shows **class loading, zlib `inflate`, etc.** (jar reads) and compiler-arena churn; **steady-state diff is net-negative** (live native heap stable/shrinking) while process RES keeps rising

## Methodology (how we're analyzing)
1. **RSS-over-time sampling** of the Logstash PID (`/proc//status` `VmRSS`) to characterize the growth/sawtooth and compare configurations.
2. **Allocator identification**: `/proc//maps`, `LD_PRELOAD`/`MALLOC_*` env to confirm glibc vs jemalloc.
3. **jemalloc heap profiling** (`MALLOC_CONF=prof:true,lg_prof_sample,lg_prof_interval`) with periodic dumps, then **`jeprof --base` diffs** to isolate *retained* (live) growth and its native call sites (e.g., `deflate`/`inflate`/`ssl`/`malloc`).
4. **JVM-side cross-checks**: `_node/stats` (heap committed vs used), GC counts, thread counts; optionally NMT.

## Mitigations tested
- `-XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=60000` (JDK 21 periodic native trim): **RES fluctuates and trends down** — effective.
- **jemalloc** via `LD_PRELOAD` (returns memory to OS more aggressively than glibc): UNDER EVALUATION.
- `MALLOC_ARENA_MAX` / periodic `malloc_trim` (`System.trim_native_heap`): known to reclaim RES.

## Reproduction (containerized)
1. `docker.elastic.co/logstash/logstash:8.19.13` on Linux/glibc, `-Xms1g -Xmx1g`.
2. Pipeline: a generator/network input + an output that exercises native paths (e.g., `s3` with `encoding => "gzip"`); API over TLS; Metricbeat scraping `:9600` every 10s.
- (S3 creds/bucket redacted: `access_key_id`/`secret_access_key`/`bucket` provided via env, e.g. `s3://`.)
3. Sample `VmRSS` over time; observe RES ratchet on glibc.
4. Compare against the same image with jemalloc preloaded and/or `TrimNativeHeapInterval` enabled.

SEE THE BELOW ANALYSIS COMMENT - https://github.com/elastic/logstash/issues/19248#issuecomment-4812075305