fluvio-community / fluvio-community/fluvio-client-python

Performance degradation on python producer.send with large number of records

Open
#258 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
19
Forks
11
PR merge metrics
No merged PRs in 30d

Description

slack thread: https://infinyon.slack.com/archives/CQ8UX6SBH/p1684349700779719

simple producer in python:
```python
#!/usr/bin/env python3
from datetime import datetime
from fluvio import Fluvio
import time

PARTITION = 0

if __name__ == "__main__":
# Connect to cluster
fluvio = Fluvio.connect()

producer = fluvio.topic_producer("simple")

start_time = time.time()

for x in range(10000):
producer.send_string("{}: timestamp: {}".format(x, datetime.now()))

# Flush the last entry
producer.flush()
print("completed in {} seconds", (time.time()-start_time))
```
10K records pushed in 1.2153611183166504 seconds, when if you increase the range to 100000
python3 producer100K.py
The same code takes 64.67606687545776 to complete.
Rust producer doesn't have such behaviour.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.