netty / netty/netty

Netty 4.1.115 Openssl Host memory getting leaked over the time

Open
#14,565 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
35.1k
Forks
16.3k
Avg merge
1d 5h
Merged PRs (30d)
143

Description

Expected behavior

Host memory should not get drained over time when using netty with mtls handshake and openssl as provider

Actual behavior

Host memory is slowly getting drained and when i ran leak checks and verified heap dumps(java) i didnt see any issue in leak suspect report but i do see issue is only with host memory not with heap. This issue was started when i switched from JDK to openssl. I am using openssl 1.0.2k. I tried checking the same with netty-boring-ssl-static. Both are resulting in draining out host memory

I ran Jemalloc to profile the host memory usage of netty application, i see hughe amount of memory is being consumed by get client certificate node. Can anyone help me understand what is the issue with that function, i checked netty-tcnative code for that function i see it is returning a buffer and being used in session.handshakefinished.

session.handshakeFinished(SSL.getSessionId(ssl), SSL.getCipherForSSL(ssl), SSL.getVersion(ssl),
SSL.getPeerCertificate(ssl), SSL.getPeerCertChain(ssl),
SSL.getTime(ssl) * 1000L, parentContext.sessionTimeout() * 1000L);

Also i enabled leak check keeping the level at PARANOID but didnt get any leak logs.

Steps to reproduce
Minimal yet complete reproducer code (or URL to code)

Switched from JDK to Openssl as provider and i am doing reference release of ssl context where my sslcontext will be recreated every 20min in my application. I am using mtls to validate and authorize CN fetched using getpeercertificateschain inside my handler. I am releasing the bytebuf which i get in channel read.

Netty version 4.1.115
JVM version (e.g. java -version) 11
OS version (e.g. uname -a) RHEL

openssl_netty_4 1 115_mtls

Kindly share your thoughts to unblock me from here, sorry if it is noop question but it is giving hard time to resolve.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the reported switch from JDK to OpenSSL, the mTLS handshake, and the native host-memory profile. Inspect netty-tcnative's getPeerCertChain path and its use in session.handshakeFinished, then compare heap and leak-check results with jemalloc. Done should be a reproducible case and a confirmed ownership or lifetime explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.