confluentinc / confluentinc/confluent-kafka-python

OverflowError: Python int too large to convert to C int when using confluent_kafka Avro deserializer

Open
#1,716 4 comments 0 reactions 0 assignees View on GitHub
component:schema-registry enhancement status:waiting-for-interest
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========
I'm encountering an OverflowError when attempting to deserialize messages using the **confluent_kafka **Avro deserializer in Python. Here's a simplified version of my code:

```
class ConsumerKafka:
def __init__(self, deserializer):
self.deserializer = deserializer
self.consumer = Consumer()

def decode(self, msg_value):
deserialized_data = self.deserializer(msg_value, None)
```
self.serilizer is AvroDeserializer object from onfluent_kafka.schema_registry.avro

when i call the self.deserializer(msg_value, None)

```
Traceback (most recent call last):
File "c:\Users\048115571\Documents\python\TTA\read_from_topic\modules\consumer.py", line 30, in decode
deserialized_data = self.deserializer(msg_value, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\048115571\AppData\Local\Programs\Python\Python311\Lib\site-packages\confluent_kafka\schema_registry\avro.py", line 429, in __call__
obj_dict = schemaless_reader(payload,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "fastavro\\_read.pyx", line 1142, in fastavro._read.schemaless_reader
File "fastavro\\_read.pyx", line 1169, in fastavro._read.schemaless_reader
File "fastavro\\_read.pyx", line 748, in fastavro._read._read_data
File "fastavro\\_read.pyx", line 621, in fastavro._read.read_record
File "fastavro\\_read.pyx", line 740, in fastavro._read._read_data
File "fastavro\\_read.pyx", line 558, in fastavro._read.read_union
File "fastavro\\_read.pyx", line 724, in fastavro._read._read_data
File "fastavro\\_read.pyx", line 393, in fastavro._read.read_array
File "fastavro\\_read.pyx", line 748, in fastavro._read._read_data
File "fastavro\\_read.pyx", line 621, in fastavro._read.read_record
File "fastavro\\_read.pyx", line 740, in fastavro._read._read_data
File "fastavro\\_read.pyx", line 558, in fastavro._read.read_union
File "fastavro\\_read.pyx", line 770, in fastavro._read._read_data
File "fastavro\\_logical_readers.pyx", line 22, in fastavro._logical_readers.read_timestamp_millis
File "fastavro\\_logical_readers.pyx", line 24, in fastavro._logical_readers.read_timestamp_millis
OverflowError: Python int too large to convert to C int
```

How to reproduce
================

confluent_avro 1.8.0
confluent-kafka 2.3.0
fastavro 1.9.2
kafka-python 2.0.2

- [ ] Operating system: windows
Checklist
=========
Please provide the following information:

- [ ] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`):
- [ ] Apache Kafka broker version:
- [ ] Client configuration: `{...}`
- [ ] Operating system:
- [ ] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [ ] Critical issue

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.