eclipse-ee4j / eclipse-ee4j/tyrus

tyrus-client 1.16: Failed SSL handshake leads to memory leak

Open
#691 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
128
Forks
49
PR merge metrics
No merged PRs in 30d

Description

I'm using the tyrus client with jdk container under JDK8.
When establishing a connection via wss, in case there is a problem with the presented certificate (in my case CertificateNotYetValidException), after some time the application throws an OutOfMemory error.

Digging deeper in the heap dump, there are a lot of instances to java.security.cert.TrustAnchor and of org.glassfish.tyrus.core.coder.CoderWrapper.
Actually, for each invocation to ClientManager#connectToServer() a whole object graph remains behind and is never garbage collected once the CertificateNotYetValidException error is thrown.

After some investigations, the problem seems to be from org.glassfish.tyrus.container.jdk.client.SslFilter.
Once the error is thrown, handleSslError() will bubble up the error to ClientFilter, but on it's way down, when hitting TaskQueueFilter::close(), it stops there.
The TaskQueueFilter will queue a close() task for SslFilter but as the taskLock is already taken (doing the handshake), the task itself is never run.
Thus, SslFilter:close() is never called and all it's resources together with the underlying TransportFilter resources are never released.

One way to fix this would be to simply add to SslFilter::handleSslError() to close() after bubbling the error:
private void handleSslError(Throwable t) {
onError(t);
close();
}
After applying this hotfix, the memory leak was solved.
Must be checked if there are no other code paths that would have other undesired side-effects.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.