4paradigm / 4paradigm/OpenMLDB
load data from parquet, poor performance for big datasets
Open
batch-engine
bug
high-priority
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 331
- Avg merge
- 12d 12h
- Merged PRs (30d)
- 1
Description
parquet file is 3.5G, and possibly 35G in memory. load data from parquet into online storage, spark bootstrap the job, starting 30min and fails of OOM.
Contributor guide
Research direction
The issue mentions loading a large Parquet file (3.5G, potentially 35G in memory) via Spark into OpenMLDB, with OOM failures after 30 minutes. Start by examining the Spark integration code and memory management for data ingestion. Look for configuration options or limits on chunk size or streaming reads. Determine what 'done' looks like by verifying the data loads without OOM and within a reasonable time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100