confluentinc / confluentinc/confluent-kafka-python

segfault when trying to construct AvroProducer

Open

#769 opened on Jan 22, 2020 0 comments 0 reactions 0 assignees View on GitHub

bug component:schema-registry investigate further

Dominant language: Python
Stars: 509
Forks: 964
Avg merge: 2d 2h
Merged PRs (30d): 14

Description

Description
===========
I get a SIGSEGV crashing my entire application when trying to construct an AvroProducer. I'm unclear if there is a problem with my configuration or not, but a segfault is not really acceptable. Is it possible to use cysignals sig_on() and sig_off to convert this to an exception?

How to reproduce
================

The following code produces the output below

avro_schema_options = getConfigProperties('avro.*', section=section_name)
# If there's any avro schema options provided, assume they're all provided and setup the avro producer
default_key_schema = avro.load(PERF_KEY_SCHEMA_PATH)
default_value_schema = avro.load(PERF_VALUE_SCHEMA_PATH)

avro_producer_params = {key[len('avro.'):]: value for key, value in avro_schema_options.items()}

self.logger.debug("Setting up AvroProducer with %r", avro_producer_params)

try:
producer = avro.AvroProducer(avro_producer_params,
default_key_schema=default_key_schema,
default_value_schema=default_value_schema)
except:
self.logger.warning("Failed to configure avro producer", exc_info=True)
else:
return producer

> Setting
> ----------
> 0 signals.so
> 1 signals.so
> 2 signals.so
> 3
> 4 ???
> 5 cimpl.so
> 6 Python
> 7 Python
> 8 Python
> 9 Python
> 10 Python
> 11 Python
> 12 Python
> 13 Python
> 14 Python
> 15 Python
> 16 Python
> 17 Python
> 18 Python
> 19 Python
> 20 Python
> 21 Python
> 22 Python
> 23 Python
> 24
> 25
> 26
> 27
> 28 Python
> 29 Python
> 30 Python
> 31 Python
> 32 Python
> 33 Python
> 34 Python
> 35 Python
> 36 Python
> 37 Python
> 38 Python
> 39 Python
> 40 Python
> 41 Python
> 42 Python
> 43 Python
> 44 Python
> 45 Python
> 46 Python
> 47 Python
> 48 Python
> 49 Python
> 50 Python
> 51 Python
> 52 Python
> 53 Python
> 54 Python
> 55 Python
> 56 Python
> 57 Python
> 58 Python
> 59 Python
> 60 Python
> 61 Python
> 62 Python
> 63 Python
> 64 Python
> 65 Python
> 66 Python
> 67 Python
> 68 Python
> 69 Python
> 70 libdyld.dylib
> ----------
> Unhandled
> This
> in
> Python will now terminate.
> ---------- up AvroProducer with {u'bootstrap.servers': u'gpu-test4.sf.smle.co:9092', u'schema.registry.url': u'http://gpu-test4.sf.smle.co:8081', u'queue.buffering.max.ms': 10} -------------------------------------------------------------- 0x0000000103f4537b print_backtrace + 65 0x0000000103f48c90 sigdie + 60 0x0000000103f48c14 cysigs_signal_handler + 351 libsystem_platform.dylib 0x00007fff614efb5d _sigtramp + 29 0x00007ffeeeaa0ac0 0x0 + 140732902542016 0x0000000103f5bb1d Producer__len__ + 13 0x00007fff3f31ae28 _PyBuiltin_Init + 8463 0x00007fff3f31ef74 PyEval_EvalFrameEx + 4753 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f323d4c _PyEval_SliceIndex + 490 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f323d4c _PyEval_SliceIndex + 490 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f2c52ea PyFunction_SetClosure + 779 0x00007fff3f2a8078 PyObject_Call + 97 0x00007fff3f2b2695 PyMethod_New + 1169 pydevd_cython_darwin_27_64.so 0x0000000101bafbd0 __pyx_pw_14_pydevd_bundle_26pydevd_cython_darwin_27_64_9PyDBFrame_5do_wait_suspend + 224 pydevd_cython_darwin_27_64.so 0x0000000101b9a69c __pyx_f_14_pydevd_bundle_26pydevd_cython_darwin_27_64_9PyDBFrame_trace_dispatch + 25660 pydevd_cython_darwin_27_64.so 0x0000000101bae8ae __pyx_pw_14_pydevd_bundle_26pydevd_cython_darwin_27_64_9PyDBFrame_21trace_dispatch + 142 pydevd_cython_darwin_27_64.so 0x0000000101bc5e8a __pyx_pw_14_pydevd_bundle_26pydevd_cython_darwin_27_64_15SafeCallWrapper_3__call__ + 122 0x00007fff3f2a8078 PyObject_Call + 97 0x00007fff3f323710 PyEval_CallObjectWithKeywords + 159 0x00007fff3f342f4b PySys_WriteStdout + 2646 0x00007fff3f342fb3 PySys_WriteStdout + 2750 0x00007fff3f31fc38 PyEval_EvalFrameEx + 8021 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f2c52ea PyFunction_SetClosure + 779 0x00007fff3f2a8078 PyObject_Call + 97 0x00007fff3f2b2695 PyMethod_New + 1169 0x00007fff3f2a8078 PyObject_Call + 97 0x00007fff3f2ed603 _PyObject_SlotCompare + 5547 0x00007fff3f2e8dcf _PyType_Lookup + 1372 0x00007fff3f2a8078 PyObject_Call + 97 0x00007fff3f31e9d3 PyEval_EvalFrameEx + 3312 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f323d4c _PyEval_SliceIndex + 490 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f31d499 PyEval_EvalCode + 32 0x00007fff3f33c645 PyParser_ASTFromFile + 287 0x00007fff3f33c6ec PyRun_FileExFlags + 130 0x00007fff3f319ea0 _PyBuiltin_Init + 4487 0x00007fff3f31e9e6 PyEval_EvalFrameEx + 3331 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f323d4c _PyEval_SliceIndex + 490 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f323cee _PyEval_SliceIndex + 396 0x00007fff3f31e498 PyEval_EvalFrameEx + 1973 0x00007fff3f31dab2 PyEval_EvalCodeEx + 1555 0x00007fff3f31d499 PyEval_EvalCode + 32 0x00007fff3f33c645 PyParser_ASTFromFile + 287 0x00007fff3f33c6ec PyRun_FileExFlags + 130 0x00007fff3f33c26b PyRun_SimpleFileExFlags + 718 0x00007fff3f34dbfa Py_Main + 3174 0x00007fff613043d5 start + 1 -------------------------------------------------------------- SIGSEGV: A segmentation fault occurred. probably occurred because a *compiled* module has a bug it and is not properly wrapped with sig_on(), sig_off(). --------------------------------------------------------------

Checklist
=========
- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`): confluent-kafka[avro]==1.2.0, 16908288
- [ ] Apache Kafka broker version:
- [ ] Client configuration: `{...}`
- [x] Operating system: Mac OS 10.14.6, CentOS 7
- [x] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [x] Critical issue

I don't know how to find the missing information. Please explain how to fill in the missing details.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

segfault when trying to construct AvroProducer

Description

Contributor guide

Assessment

Get new issues in your inbox