4paradigm / 4paradigm/OpenMLDB

load data: load parquet to online, unable to ensure correctness

Open
#3,424 0 comments 0 reactions 2 assignees Claimed by @vagetablechicken View on GitHub
batch-engine bug high-priority
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

load data can't ensure correctness: insert more rows to online than parquet file

Steps:
1. load data from parquet ( 600 million rows, 3.5 G parquet file in disk, 300 parquet files total )
2. with spark local, driver memory: 64G, max task in parallel: 8
3. spark will split job into 50 tasks, and 8 tasks one time
4. tasks might fail due to `IOException`, rpc to tablet server timeout of 20s
5. spark sees to retry the whole job (50 tasks), result in insert data from parquet double times

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.