apache / apache/pulsar-client-go
DisableBatching still go thru the batching logic and AUTO_CONSUME can not consume with correct schema type
- Dominant language
- Go
- Stars
- 745
- Forks
- 389
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 3
Description
#### Actual behavior
When using go client to produce messages with avro schema, and consume using
```
pulsar-client consume -st auto_consume -s test topic_name -n 0
```
I'm getting, messages with string schema
```
----- got message -----
key:[Ml8=], properties:[], content:{value=, type=class java.lang.String}
----- got message -----
key:[MjNf], properties:[], content:{value=..@, type=class java.lang.String}
```
This is with or without `DisableBatching` enabled.
#### Expected behavior
This is inconsistent with the python client.
When using python, and batching is enabled, I get the above string typed message.
When batching is disabled, I get avro typed message.
I'm not sure how batching has anything to do with `AUTO_CONSUME`'s way of inferring schema type. I expect, whether batching enabled or not, I can always get avro typed messages.
#### Steps to reproduce
1. `pulsar-client consume -st auto_consume -s test topic_name -n 0`
2. use go client to produce avro messages with `DisableBatching=true` or `DisableBatching=false`
It looks like even if `DisableBatching` is set to true, it still uses a `batchBuilder` to build a batch, and that `batchBuilder` respects all the other batching related configs.
https://github.com/apache/pulsar-client-go/blob/efb102522f7c8b25d13c282512fa5a5fe2f2ae44/pulsar/producer_partition.go#L246
Possibly related issue: https://github.com/apache/pulsar/issues/11288
#### System configuration
**Pulsar version**: 2.9.0
Contributor guide
Research direction
Start at pulsar/producer_partition.go around line 246 and reproduce with the Go producer using DisableBatching=true and false, then consume with the documented pulsar-client AUTO_CONSUME command. Trace how batchBuilder is selected and how the produced Avro schema reaches AUTO_CONSUME. Done means disabling batching no longer follows batching behavior and AUTO_CONSUME reports the correct Avro-typed messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100