apache / apache/pulsar

Avro schema incompatible with schema upload vs python producer

Open
#9,571 11 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
Hi i just want to know if i do something wrong or if someone has identify following behavior (and found some solution for fixing it)
I create a topic and upload the Avro Schema as Yaml
```
pulsar/bin/pulsar-admin topics create persistent://schema/test3/foo22
/pulsar/bin/pulsar-admin schemas upload persistent://schema/test3/foo22 -f /git_files/schema/schema-validation.avro
```
Content of Avro Yaml Schema:
`{"type": "AVRO","schema":"{\"name\":\"Validation\",\"type\":\"record\",\"fields\":[{\"name\":\"messagetimestamp\",\"type\":\"string\"}]}","properties": {}}`

Schema looks like that with schemas get
```/pulsar/bin/pulsar-admin schemas get schema/test3/foo22
{
"version": 0,
"schemaInfo": {
"name": "foo22",
"schema": {
"name": "Validation",
"type": "record",
"fields": [
{
"name": "messagetimestamp",
"type": "string"
}
]
},
"type": "AVRO",
"properties": {}
}
}
```
set-is-allow-auot-update-schema is set to disable(!) to cover that nowone is “destroying” or topic
but if i know use a python producer the schema is failing with “incompatible schema”
python code:
```
class Validation(Record):
messagetimestamp = String(required=True)
producer = client.create_producer(topic, schema=AvroSchema(Validation))
inputString='Test55'
producer.send(Validation(
messagetimestamp = inputString
))
print("Message send")
client.close()
```
same behaviour with python pulsar functions .
if i turn on the auto schema update a new schema is created but total same structure:

```
{
"version": 1,
"schemaInfo": {
"name": "foo22",
"schema": {
"name": "Validation",
"type": "record",
"fields": [
{
"name": "messagetimestamp",
"type": "string"
}
]
},
"type": "AVRO",
"properties": {}
}
}
```
Any idea why that happens and how can i avoid that multiple producer (written in Java, Python) are changing the schema all time?!
**To Reproduce**
Steps to reproduce the behavior:
1. create topic
2. upload avro Schema (with yml reference -f yml file)
3. deactivate auto schema update
4. create python consumer (incompatible schema)
5. enable auto schema update
6. check schema (schema version changed)

**Expected behavior**
Schema version shouldn't be updated if python schema is the same as given in the schema upload (could it be that there is some hidden character given?!)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the listed steps with pulsar-admin schemas upload/get and the Python AvroSchema producer, comparing the uploaded schema with the producer-generated schema. Check whether the schemas are treated as incompatible and why auto-update creates a new version; done means equivalent Java and Python producer schemas are accepted without an unnecessary version change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.