apache / apache/pulsar

Client should not retry when TLS authentication is configured

Open
#22,026 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

When I set up a standalone that enables mTLS authentication according to the [official document](https://pulsar.apache.org/docs/next/security-tls-transport/), and then I didn't configure authentication at client side, the client kept attempting to reconnect.

Client code:

```java
final var rootPath = "/path/to/certs/";
@Cleanup var client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://localhost:6651/")
.tlsTrustCertsFilePath(rootPath + "/ca.cert.pem")
// Uncommenting the following two lines will work
//.authentication("org.apache.pulsar.client.impl.auth.AuthenticationTls",
// "tlsCertFile:" + rootPath + "client.cert.pem,tlsKeyFile:" + rootPath + "client.key-pk8.pem")
.build();
@Cleanup var producer = client.newProducer().topic("java-topic").create();
producer.send("hello".getBytes());
```

Client logs:

```
[2024-02-05 17:54:05,409] pulsar-client-io-1-1id WARN [localhost/127.0.0.1:6651] Got exception io.netty.handler.codec.DecoderException: io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslException: error:1000045c:SSL routines:OPENSSL_internal:TLSV1_ALERT_CERTIFICATE_REQUIRED
(stacks...)
Caused by: io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslException: error:1000045c:SSL routines:OPENSSL_internal:TLSV1_ALERT_CERTIFICATE_REQUIRED
(stacks...)
[2024-02-05 17:54:05,409] pulsar-client-io-1-1id INFO [id: 0x620e0cac, L:/127.0.0.1:60237 ! R:localhost/127.0.0.1:6651] Disconnected (org.apache.pulsar.client.impl.ClientCnx)
[2024-02-05 17:54:05,409] pulsar-client-io-1-1id WARN [[id: 0x620e0cac, L:/127.0.0.1:60237 ! R:localhost/127.0.0.1:6651]] Connection handshake failed: org.apache.pulsar.client.api.PulsarClientException: Connection already closed (org.apache.pulsar.client.impl.ConnectionPool)
```

Broker logs:

```
2024-02-05T17:54:05,409+0800 [pulsar-io-18-1] WARN org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:60237] Got exception io.netty.handler.codec.DecoderException: io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.newSSLExceptionForError(ReferenceCountedOpenSslEngine.java:1377)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.needWrapAgain(ReferenceCountedOpenSslEngine.java:1363)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1394)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1325)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1426)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1469)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:223)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1257)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1297)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
... 21 more
```

### Solution

Since the connection could not be established before a SSL handshake succeeds, it seems impossible to return an error response from the broker side. So it could only be handled at the client side.

### Alternatives

_No response_

### Anything else?

The behavior of the C++ client is more friendly than the Java client.

```c++
ClientConfiguration config;
std::string rootDir = "/path/to/certs/";
config.setUseTls(true); // shouldn't be needed soon
config.setTlsTrustCertsFilePath(rootDir + "ca.cert.pem");
config.setTlsAllowInsecureConnection(false);
//config.setAuth(pulsar::AuthTls::create(rootDir + "client.cert.pem", rootDir + "client.key-pk8.pem"));
config.setValidateHostName(false);
Client client("pulsar+ssl://localhost:6651", config);
```

Error logs:

```
2024-02-05 18:13:10.212 INFO [0x16b917000] ClientConnection:403 | [127.0.0.1:60714 -> 127.0.0.1:6651] Connected to broker
2024-02-05 18:13:10.215 ERROR [0x16b917000] ClientConnection:507 | [127.0.0.1:60714 -> 127.0.0.1:6651] Handshake failed: sslv3 alert handshake failure (SSL routines)
2024-02-05 18:13:10.215 ERROR [0x16b917000] ClientConnection:1304 | [127.0.0.1:60714 -> 127.0.0.1:6651] Connection closed with ConnectError (refCnt: 2)
```

Though the `ResultConnectError` error code might not be the best. Maybe we should fail it with `ResultAuthenticationError`.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing the Java client's TLS handshake failure and reconnect handling, then compare the behavior described for the C++ client. Done means a missing client certificate produces a terminal authentication or connection error instead of repeated retries, with coverage for the reported mTLS setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
authentication, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.