apache / apache/pulsar

[Doc] Provide context around timing for producer failover and suggested settings for sendTimeout and failoverDelay

Open
#20,793 1 comment 0 reactions 0 assignees View on GitHub
doc-required Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### What issue do you find in Pulsar docs?

https://pulsar.apache.org/docs/next/client-libraries-cluster-level-failover/#automatic-failover

Implementing the code above when using a send or sendAsync results in dataloss unless additional code changes are made. It is also unclear the order of how the send/sendAsync timeout set with sendTimeout and the failoverDelay work together.

### What is your suggestion?

1. Add an explanation of the order sendTimeout and failoverDelay are implemented. It appears the sendTimeout must first elapse with messages being removed from the async queue without being acked, resulting in possible dataloss unless the developer resends the same messages with sendAsync. For sendAsync, this is probably an unreasonable request to have the developer keep track of which messages timed out. Only after the sendTimeout elapses, does failoverDelay kick in resulting in a cluster lever failover. Making failoverDelay greater or less than sendTimeout does not change this behavior.
2. The alternate approach is to set sendTimeout to 0, which does not result in any dataloss in my tests but is not mentioned in the doc. Right now the docs only show how to build the client, but not the producer or sending a message with sendAsync. We should at a minimum include an example where we also create the producer and set the sendTimeout to 0, along with an explanation on how this affects the order of events during a failover (resulting in no dataloss) and any other implications of setting this to zero.
3. This part of the code does not compile since the builder has additional required fields.
PulsarClient pulsarClient = PulsarClient.builder()
.build();
4. Add an explanation for when failover.initialize(pulsarClient) is needed. It's unclear when this step is needed (or if it's needed at all except maybe if your're changing client settings after the initial build?).
5. Explain that other authorization schemes are possible besides tls with certs. For example, I was able to use OAuth2 by using AuthenticationFactoryOAuth2 instead of AuthenticationFactory, while also omitting secondaryTlsTrustCertsFilePath from builder() for AutoClusterFailover.

### Any reference?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Review the automatic-failover section at the linked documentation page, including PulsarClient.builder(), sendTimeout, failoverDelay, failover.initialize(pulsarClient), and the AutoClusterFailover authentication settings. Reproduce the documented example and verify the producer/sendAsync behavior around failover. Done means the page compiles, explains the event order and settings, and covers producer construction and alternative authorization schemes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.