confluentinc / confluentinc/confluent-kafka-python

AvroDeserializer reader schema cannot ignore values required in writer schema

Open
#887 1 comment 4 reactions 0 assignees View on GitHub
component:schema-registry component:serdes good first issue
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========

I'm using `AvroDeserializer` to deserialize records from a topic and I'm running into issues when confluent-kafka-python calls fastavro in the deserialization function:

```python
obj_dict = schemaless_reader(payload,
writer_schema,
self._reader_schema)
```
([link to the source code line](https://github.com/confluentinc/confluent-kafka-python/blob/6ac115fcc886e609e30d3727868c2d52310e24c1/confluent_kafka/schema_registry/avro.py#L321))

The `reader_schema` that I pass in omits a field that is present in the `writer_schema` because I'm not interested in reading it. However, this results in fastavro raising the following exception:

```bash
File "fastavro/_read.pyx", line 835, in fastavro._read.schemaless_reader
File "fastavro/_read.pyx", line 846, in fastavro._read.schemaless_reader
File "fastavro/_read.pyx", line 561, in fastavro._read._read_data
File "fastavro/_read.pyx", line 472, in fastavro._read.read_record
File "fastavro/_read.pyx", line 559, in fastavro._read._read_data
File "fastavro/_read.pyx", line 413, in fastavro._read.read_union
File "fastavro/_read.pyx", line 73, in fastavro._read.match_types
File "fastavro/_read.pyx", line 127, in fastavro._read.match_schemas
fastavro._read_common.SchemaResolutionError: Schema mismatch: {'avro.java.string': 'String', 'type': 'string'} is not null
```

I suppose this makes sense from fastavro point-of-view because the purpose of the `schemaless_reader` `reader_schema` is for "schema migration" ([see here](https://github.com/fastavro/fastavro/blob/7a5220b30c4bbe616a1a42f62fff118cca0928d2/fastavro/_read_py.py#L875)) and a required field cannot just be dropped. From a Kafka topic reader's point of view, it doesn't make sense as a reader might only be interested in a subset of data.

Personally, my issue would be solved by allowing `AvroDeserializer` to allow a `None` value for the `reader_schema` and in that case ignoring `reader_schema` completely by calling fastavro like this:

```python
obj_dict = schemaless_reader(payload,
writer_schema,
None)
```

Or do you feel this should be addressed with fastavro?

How to reproduce
================

Checklist
=========
Please provide the following information:

- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`): latest (1.4.2)
- [x] Apache Kafka broker version: Confluent Cloud
- [ ] Client configuration: `{...}`
- [x] Operating system: Ubuntu 18.04 (under WSL1)
- [ ] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [ ] Critical issue

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.