TLS mutual Authentication with keystore configuration on Linux OS fails to establish a successful connection
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Java Client(2.11.0)-->broker(2.11.0)
Broker instance OS: Amazon Linux 2023
Java App running OS : RHEL and Ubuntu
### Minimal reproduce step
Configure the Pulsar instance for mTLS authentication using Keystore. Then create the Pulsar client using the following sample code.
```java
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://broker.example.com:6651/")
.useKeyStoreTls(true)
.tlsTrustStorePath("/var/private/tls/client.truststore.jks")
.tlsTrustStorePassword("clientpw")
.allowTlsInsecureConnection(false)
.enableTlsHostnameVerification(false)
.authentication(
"org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
"keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
.build();
```
### What did you expect to see?
It should get succeeded with the correct keystore path configuration for Linux.
### What did you see instead?
```
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
2023-05-03T11:28:07,645 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2023-05-03T11:28:07,646 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2023-05-03T11:28:08,252 DEBUG [pulsar-client-io-1-1] org.apache.pulsar.common.util.SecurityUtility - Already instantiated Bouncy Castle provider BCFIPS
2023-05-03T11:28:08,299 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-05-03T11:28:08,304 WARN [pulsar-client-io-1-1] org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 : java.lang.NullPointerException
2023-05-03T11:28:08,407 WARN [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.PulsarClientImpl - [topic: persistent://public/default/test-topic] Could not get connection while getPartitionedTopicMetadata -- Will try again in 100 ms
2023-05-03T11:28:08,407 DEBUG [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.ConnectionPool - Connection for ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 not found in cache
2023-05-03T11:28:08,410 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
```
### Anything else?
As you can observe the debug logs, the keystore path we are providing is not taking into effect and it is being set to null and throwing the file not found error.
**Note**: Same source code is working fine with windows machine.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start at KeyStoreSSLContext.createSSLContext (line 145) and createClientKeyStoreSslContext (line 230), then trace configuration from AuthenticationKeyStoreTls through NettySSLContextAutoRefreshBuilder.update. Reproduce the Linux keystore setup and verify that the configured truststore and keystore paths are used without the null FileNotFoundException and that the mTLS connection succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux
- Domain
- authentication, backend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100