4paradigm / 4paradigm/OpenMLDB
load data from parquet, poor performance for big datasets
Ouverte
#3,396
2 commentaires
0 réactions
2 personnes assignées
Réclamée par @vagetablechicken
Voir sur GitHub
batch-engine
bug
high-priority
- Langage dominant
- C++
- Étoiles
- 1.7k
- Forks
- 331
- Merge moyen
- 12 j 12 h
- PR mergées (30 j)
- 1
Description
parquet file is 3.5G, and possibly 35G in memory. load data from parquet into online storage, spark bootstrap the job, starting 30min and fails of OOM.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
The issue mentions loading a large Parquet file (3.5G, potentially 35G in memory) via Spark into OpenMLDB, with OOM failures after 30 minutes. Start by examining the Spark integration code and memory management for data ingestion. Look for configuration options or limits on chunk size or streaming reads. Determine what 'done' looks like by verifying the data loads without OOM and within a reasonable time.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- spark
- Domaine
- data-engineering, databases, performance
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 30/100