fluvio-community / fluvio-community/fluvio-docs

Provide Troubleshooting Entry for Excedeed Storage

Open
#83 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
4
Forks
11
PR merge metrics
No merged PRs in 30d

Description

From Alexanders Mikhalev issue https://infinyon.slack.com/archives/CN1D204C9/p1720005955323709

Context:

Hey team, I am testing sql to fluvio prototype for Klarian (written by Bryony), but I can't
write more then 14000000 records into fluvio topic, I have a local fluvio cluster on mac
book m3 max, it shall have enough capacity. It's strange behaviour: python script just hangs,
adding batching or multithreading didn't help: code is https://gist.github.com/AlexMikhalev/8b5ee7f284f99b75f80f7b912117cfb9
I have modification which just dumps all records into parquet file locally (in batches) and it works fine. Any hints how to debug?

I am currently on a "smaller" sql dump which is ~5GB, my total intake is sqlite 90GB file.

---

A Troubleshoot entry should be added as part of new docs to tackle this issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.