eclipse-ee4j / eclipse-ee4j/tyrus
tyrus-client 1.16: Failed SSL handshake leads to memory leak
- Dominant language
- Java
- Stars
- 128
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
I'm using the tyrus client with jdk container under JDK8.
When establishing a connection via wss, in case there is a problem with the presented certificate (in my case CertificateNotYetValidException), after some time the application throws an OutOfMemory error.
Digging deeper in the heap dump, there are a lot of instances to java.security.cert.TrustAnchor and of org.glassfish.tyrus.core.coder.CoderWrapper.
Actually, for each invocation to ClientManager#connectToServer() a whole object graph remains behind and is never garbage collected once the CertificateNotYetValidException error is thrown.
After some investigations, the problem seems to be from org.glassfish.tyrus.container.jdk.client.SslFilter.
Once the error is thrown, handleSslError() will bubble up the error to ClientFilter, but on it's way down, when hitting TaskQueueFilter::close(), it stops there.
The TaskQueueFilter will queue a close() task for SslFilter but as the taskLock is already taken (doing the handshake), the task itself is never run.
Thus, SslFilter:close() is never called and all it's resources together with the underlying TransportFilter resources are never released.
One way to fix this would be to simply add to SslFilter::handleSslError() to close() after bubbling the error:
private void handleSslError(Throwable t) {
onError(t);
close();
}
After applying this hotfix, the memory leak was solved.
Must be checked if there are no other code paths that would have other undesired side-effects.
Contributor guide
Assessment
This issue has not been assessed yet.