apache / apache/pulsar

Avro "fixed" field causes ClassCastException

Open
#9,718 3 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
If you have an Avro "fixed" field in a record (or presumably if you try to produce a "fixed" field directly), you get a ClassCastException when producing a message.

```
java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')
```

(full traceback below)

I slightly tweaked @sijie 's [`pulsar-avro-schema-example`](https://github.com/sijie/pulsar-avro-schema-example) project to demonstrate the issue: [patricklucas/pulsar-avro-schema-example](https://github.com/patricklucas/pulsar-avro-schema-example).

(I updated dependencies, and then added a [fixed field](https://github.com/patricklucas/pulsar-avro-schema-example/blob/master/src/main/avro/fixed.avsc) to the [TwitterSchema](https://github.com/patricklucas/pulsar-avro-schema-example/blob/master/src/main/avro/twitter.avsc) record. Additionally, I added code to serialize the message to JSON before producing, to demonstrate that Avro itself handles the message correctly)

It's possible this is unsupported in some way by Pulsar, but I couldn't find anything in the documentation about such limitations, so I thought I'd ask here.

**To Reproduce**
Steps to reproduce the behavior:
1. Clone the project [patricklucas/pulsar-avro-schema-example](https://github.com/patricklucas/pulsar-avro-schema-example)
2. Run [TweetProducer](https://github.com/patricklucas/pulsar-avro-schema-example/blob/master/src/main/java/org/apache/pulsar/examples/TweetProducer.java), passing a Pulsar service URL as a single argument
3. Observe error

**Expected behavior**
The message should be serialized successfully.

**Additional context**

Full traceback:

```
org.apache.pulsar.client.api.SchemaSerializationException: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')
at org.apache.pulsar.client.impl.schema.writer.AvroWriter.write(AvroWriter.java:55)
at org.apache.pulsar.client.impl.schema.AbstractStructSchema.encode(AbstractStructSchema.java:50)
at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.value(TypedMessageBuilderImpl.java:178)
at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:63)
at org.apache.pulsar.examples.TweetProducer.main(TweetProducer.java:43)
Caused by: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeFixed(GenericDatumWriter.java:339)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:147)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:206)
at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:186)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:195)
at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:130)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at org.apache.pulsar.client.impl.schema.writer.AvroWriter.write(AvroWriter.java:53)
... 4 more
```

Contributor guide

Open the contributing guide

Research direction

Start with AvroWriter.java at the failing write call in the traceback, then reproduce the problem with TweetProducer from the linked pulsar-avro-schema-example project. Compare the fixed.avsc and twitter.avsc schemas and the serialization path; done means a record containing an Avro fixed field is serialized and produced successfully without the ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.