4paradigm / 4paradigm/OpenMLDB
load data from parquet, poor performance for big datasets
Abierto
batch-engine
bug
high-priority
- Lenguaje dominante
- C++
- Estrellas
- 1.7k
- Forks
- 331
- Merge medio
- 12 d 12 h
- PR fusionados (30 d)
- 1
Descripción
parquet file is 3.5G, and possibly 35G in memory. load data from parquet into online storage, spark bootstrap the job, starting 30min and fails of OOM.
Guía de contribución
Línea de trabajo
The issue mentions loading a large Parquet file (3.5G, potentially 35G in memory) via Spark into OpenMLDB, with OOM failures after 30 minutes. Start by examining the Spark integration code and memory management for data ingestion. Look for configuration options or limits on chunk size or streaming reads. Determine what 'done' looks like by verifying the data loads without OOM and within a reasonable time.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- spark
- Área
- data-engineering, databases, performance
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 30/100