apache / apache/pulsar-dotpulsar

Message loss with message delivery not guaranteed

Open
#277 20 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
271
Forks
87
PR merge metrics
No merged PRs in 30d

Description

### Description

I have a pulsar client created with a one producer created for a topic and ran a load tests and I could see that few of the messages are lost. In the current implementation is message delivery guaranteed with a publisher confirmation ?

Pulsar broker :

Current version of pulsar is: 4.1.0

Broker Config

[broker.conf.txt](https://github.com/user-attachments/files/22680110/broker.conf.txt)

Library Version: 4.3.2

**Client and Producer code**

builder = PulsarClient.Builder()
.Authentication(AuthenticationFactory.Token(TOKEN))
.CheckCertificateRevocation(true)
.ExceptionHandler(new CustomExceptionHandler())
.KeepAliveInterval(new TimeSpan(0, 0, 30))
.RetryInterval(new TimeSpan(0, 0, 2))
.CloseInactiveConnectionsInterval(new TimeSpan(0, 0, 30));
builder.ConnectionSecurity(EncryptionPolicy.EnforceEncrypted)
.ServiceUrl(new Uri($"pulsar+ssl://{pulsarHostName}:6651"))
.TrustedCertificateAuthority(Certificate.GenerateCertificate(certificateOption.Value.Pulsar.Ca))
.VerifyCertificateAuthority(true)
.VerifyCertificateName(false);

_client = builder.Build();
_producer = _client.NewProducer(Schema.ByteArray)
.ProducerName($"{@event}-{Environment.MachineName}")

.StateChangedHandler(producerStateChanged =>
{
var topicName = producerStateChanged.Producer.Topic;
var state = producerStateChanged.ProducerState;

_logger.LogDebug(new ApplicationLogMessage
{
Message = $"The producer for topic '{topicName}' changed state to '{state}'"
});
})
.Topic($"persistent://global/seed/Test)
.Create();

**Publisher code**

var messageId = await producer.Send(new MessageMetadata(), byteArray, CancellationToken.None).ConfigureAwait(false);

### Reproduction Steps

The publisher code is exposed via an http end point.

Run a load test

Number of users : 10
Number of iterations : 20

### Expected behavior

Expected: 200 Messages to be published.

### Actual behavior

There are few messages loss with EntryId and LedgerId returned as negative values

### Regression?

_No response_

### Known Workarounds

_No response_

### Configuration

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the publisher's producer.Send call and the attached broker.conf.txt; reproduce the 10-user, 20-iteration load test and inspect returned MessageId EntryId and LedgerId values. Compare publisher confirmations with the broker configuration and determine whether all 200 sends are acknowledged; done means the loss or negative-ID behavior is explained and covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.