4paradigm / 4paradigm/OpenMLDB
load data: load parquet to online, unable to ensure correctness
- Langage dominant
- C++
- Étoiles
- 1.7k
- Forks
- 331
- Merge moyen
- 12 j 12 h
- PR mergées (30 j)
- 1
Description
load data can't ensure correctness: insert more rows to online than parquet file
Steps:
1. load data from parquet ( 600 million rows, 3.5 G parquet file in disk, 300 parquet files total )
2. with spark local, driver memory: 64G, max task in parallel: 8
3. spark will split job into 50 tasks, and 8 tasks one time
4. tasks might fail due to `IOException`, rpc to tablet server timeout of 20s
5. spark sees to retry the whole job (50 tasks), result in insert data from parquet double times
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
The issue describes a data loading correctness problem in OpenMLDB involving Spark jobs and Parquet files. Look at the load data flow, likely in the Spark connector or ingestion module, focusing on task failure handling and retry logic. Check for idempotency mechanisms and RPC timeout configurations to tablet servers. Running a test with a smaller dataset to reproduce the duplicate insertion would be a first step.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- spark
- Domaine
- data-engineering, databases
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100