apache / apache/pulsar

TLS mutual Authentication with keystore configuration on Linux OS fails to establish a successful connection

Open
#20,249 4 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Java Client(2.11.0)-->broker(2.11.0)
Broker instance OS: Amazon Linux 2023
Java App running OS : RHEL and Ubuntu

### Minimal reproduce step

Configure the Pulsar instance for mTLS authentication using Keystore. Then create the Pulsar client using the following sample code.

```java
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://broker.example.com:6651/")
.useKeyStoreTls(true)
.tlsTrustStorePath("/var/private/tls/client.truststore.jks")
.tlsTrustStorePassword("clientpw")
.allowTlsInsecureConnection(false)
.enableTlsHostnameVerification(false)
.authentication(
"org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
"keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
.build();
```

### What did you expect to see?

It should get succeeded with the correct keystore path configuration for Linux.

### What did you see instead?

```
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
2023-05-03T11:28:07,645 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2023-05-03T11:28:07,646 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024

2023-05-03T11:28:08,252 DEBUG [pulsar-client-io-1-1] org.apache.pulsar.common.util.SecurityUtility - Already instantiated Bouncy Castle provider BCFIPS
2023-05-03T11:28:08,299 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-05-03T11:28:08,304 WARN [pulsar-client-io-1-1] org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 : java.lang.NullPointerException
2023-05-03T11:28:08,407 WARN [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.PulsarClientImpl - [topic: persistent://public/default/test-topic] Could not get connection while getPartitionedTopicMetadata -- Will try again in 100 ms
2023-05-03T11:28:08,407 DEBUG [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.ConnectionPool - Connection for ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 not found in cache
2023-05-03T11:28:08,410 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
```

### Anything else?

As you can observe the debug logs, the keystore path we are providing is not taking into effect and it is being set to null and throwing the file not found error.

**Note**: Same source code is working fine with windows machine.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at KeyStoreSSLContext.createSSLContext (line 145) and createClientKeyStoreSslContext (line 230), then trace configuration from AuthenticationKeyStoreTls through NettySSLContextAutoRefreshBuilder.update. Reproduce the Linux keystore setup and verify that the configured truststore and keystore paths are used without the null FileNotFoundException and that the mTLS connection succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
authentication, backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.