apache / apache/pulsar

[Bug] Avro serialisation problems with Pulsar client

Open
#21,230 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Pulsar latest version: 3.1.0

### Minimal reproduce step

The problem happens when using Avro schema, specifically for optional records that contain logical type.

Example:

```
record ExampleDecimal {
union{null, RecordWithDecimal} optional_record_with_decimal = null;
}

record RecordWithDecimal {
decimal(17,4) amount;
}
```

### What did you expect to see?

Sending Avro messages in Pulsar should work with any Pulsar client either shaded or unshaded.

### What did you see instead?

When using Pulsar-client with an Avro schema that contains an optional record with a logical type, an error occurs when Pulsar producer calls `AvroWtiter.write()` to serialise the message. This happens because Pulsar-client uses shade dependencies including Apache Avro shade dependency. That causes the problems because Avro calls `instance of IndexedRecord` to resolve the type of the union. Now that IndexedRecord is shaded in Pulsar so that call never returns true because the record is an instance of the normal IndexedRecord not the shaded one. That leads to a failure in serialising the record because Avro can't resolve its type.

Another problem is the error message I was getting is very misleading: "No recommended schema for decimal (scale is required)". Only after thorough debugging, I could understand what's going wrong.

### Anything else?

A workaround is to use Pulsar-client-original instead which doesn't use shade dependencies, but this issue is raised because Avro serialisation is now broken in the main/default Puslar client.

Also, documentation doesn't mention anything about the original client, I had to look at Pulsar code to find it.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the optional-record and logical-type case described in the issue, then trace Pulsar-client's call to AvroWriter.write() and compare the shaded and Pulsar-client-original paths. Done means the default client serializes the record correctly, the failure message is actionable, and the client workaround is documented if still relevant.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.