fluvio-community / fluvio-community/fluvio-docs
Provide Troubleshooting Entry for Excedeed Storage
- Dominant language
- MDX
- Stars
- 4
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
From Alexanders Mikhalev issue https://infinyon.slack.com/archives/CN1D204C9/p1720005955323709
Context:
Hey team, I am testing sql to fluvio prototype for Klarian (written by Bryony), but I can't
write more then 14000000 records into fluvio topic, I have a local fluvio cluster on mac
book m3 max, it shall have enough capacity. It's strange behaviour: python script just hangs,
adding batching or multithreading didn't help: code is https://gist.github.com/AlexMikhalev/8b5ee7f284f99b75f80f7b912117cfb9
I have modification which just dumps all records into parquet file locally (in batches) and it works fine. Any hints how to debug?
I am currently on a "smaller" sql dump which is ~5GB, my total intake is sqlite 90GB file.
---
A Troubleshoot entry should be added as part of new docs to tackle this issue.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.