4paradigm / 4paradigm/OpenMLDB

`load data` in cluster mode may cause out-of-order insertion, leading poor performance for long window

未关闭
#2,544 1 条评论 0 个 reaction 已指派 1 人 已被 @zhanghaohit 认领 在 GitHub 查看
bug
主要语言
C++
星标
1.7k
派生
331
平均合并
12 天 12 小时
30 天内合并 PR
1

描述

**Bug Description**
if `spark` is configured with multiple threads, say `local[*]` or `local[32]`, data loading will run in parallel, causing out-of-order loading.

It is ok for normal data insertion. But `long window optimization` requires that data is loaded mostly in order.

**Potential solution**
- remove the `in order loading` requirement for `long window`, by deleting the old entry and inserting the new one in the pre-aggr table.

贡献指南

打开贡献指南

调研方向

The issue relates to the 'load data' operation in cluster mode and its interaction with long window optimization. Investigate the data loading path in the codebase, particularly looking for parallel execution logic. Examine the long window optimization implementation to understand the in-order requirement. Check if there are existing tests for data loading or window performance to verify changes.

由索引模型根据 Issue 内容生成。

评估

技术栈
spark
领域
databases, performance
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。