Investigate adding official support for jemalloc on linux
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Because java relies on glibc's malloc implementation, allocations off-heap are something of a black box that can make troubleshooting memory growth difficult.
In support cases, we have found and fixed memory leaks by injecting a profiling build of `jemalloc` with `LD_PRELOAD` and using its profiling tools. There is also a general consensus that jemalloc is an improved implementation for long-lived and highly-concurrent applications with a better profile for both improved throughput and reduced overhead, so it would be good to have official support.
Consider adding opt-in support for selecting an available jemalloc on linux hosts.
Acceptance Criteria:
- environment flag like `LS_MALLOC_IMPLEMENTATION=jemalloc`;
- if explicitly set on a linux host and able to resolve the jemalloc libraries, inject them into `LD_PRELOAD` in `lib/logstash.lib.sh` _before_ invoking the jruby subprocess; else log helpfully to stderr and continue without injecting.
- run integration tests (possibly a randomised subset) with jemalloc selected
- provide a straight-forward path for using this feature through docker (notably jemalloc and jemalloc-devel are _not_ available in any enabled-by-default repos for the ubi9-minimal base image)
Additional Considerations:
- We _may_ want to support other allocators in the future, such as Microsoft's mimalloc on Windows or Google's tcmalloc on linux. The opt-in flag should be designed in such a way that it's difficult to end up with a configuration that is ambiguous if and when support for other allocators is added.
- Support on Mac is very difficult and my not be plausible since SIP is enabled by default, preventing libraries from being dynamically injected.
Contributor guide
Research direction
Start by reading lib/logstash.lib.sh and tracing how it invokes the JRuby subprocess and how Linux environment flags are handled. Review the existing integration tests and Docker configuration, including the ubi9-minimal image constraints. Done means an explicit jemalloc selection behaves as specified, failures log to stderr and continue safely, integration coverage runs with jemalloc, and Docker has a straightforward setup path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, linux, shell
- Domain
- devops, operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100