apache / apache/pulsar-client-go

DisableBatching still go thru the batching logic and AUTO_CONSUME can not consume with correct schema type

Open
#681 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Actual behavior

When using go client to produce messages with avro schema, and consume using

```
pulsar-client consume -st auto_consume -s test topic_name -n 0
```

I'm getting, messages with string schema

```
----- got message -----
key:[Ml8=], properties:[], content:{value=, type=class java.lang.String}
----- got message -----
key:[MjNf], properties:[], content:{value=..@, type=class java.lang.String}
```

This is with or without `DisableBatching` enabled.

#### Expected behavior

This is inconsistent with the python client.

When using python, and batching is enabled, I get the above string typed message.

When batching is disabled, I get avro typed message.

I'm not sure how batching has anything to do with `AUTO_CONSUME`'s way of inferring schema type. I expect, whether batching enabled or not, I can always get avro typed messages.

#### Steps to reproduce

1. `pulsar-client consume -st auto_consume -s test topic_name -n 0`
2. use go client to produce avro messages with `DisableBatching=true` or `DisableBatching=false`

It looks like even if `DisableBatching` is set to true, it still uses a `batchBuilder` to build a batch, and that `batchBuilder` respects all the other batching related configs.

https://github.com/apache/pulsar-client-go/blob/efb102522f7c8b25d13c282512fa5a5fe2f2ae44/pulsar/producer_partition.go#L246

Possibly related issue: https://github.com/apache/pulsar/issues/11288

#### System configuration
**Pulsar version**: 2.9.0

Contributor guide

Open the contributing guide

Research direction

Start at pulsar/producer_partition.go around line 246 and reproduce with the Go producer using DisableBatching=true and false, then consume with the documented pulsar-client AUTO_CONSUME command. Trace how batchBuilder is selected and how the produced Avro schema reaches AUTO_CONSUME. Done means disabling batching no longer follows batching behavior and AUTO_CONSUME reports the correct Avro-typed messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.