4paradigm / 4paradigm/OpenMLDB

`load data` in cluster mode may cause out-of-order insertion, leading poor performance for long window

Offen
#2,544 1 Kommentar 0 Reaktionen 1 zugewiesene Person Beansprucht von @zhanghaohit Auf GitHub ansehen
bug
Vorherrschende Sprache
C++
Sterne
1.7k
Forks
331
Ø Merge
12 T. 12 Std.
Gemergte PRs (30 T.)
1

Beschreibung

**Bug Description**
if `spark` is configured with multiple threads, say `local[*]` or `local[32]`, data loading will run in parallel, causing out-of-order loading.

It is ok for normal data insertion. But `long window optimization` requires that data is loaded mostly in order.

**Potential solution**
- remove the `in order loading` requirement for `long window`, by deleting the old entry and inserting the new one in the pre-aggr table.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue relates to the 'load data' operation in cluster mode and its interaction with long window optimization. Investigate the data loading path in the codebase, particularly looking for parallel execution logic. Examine the long window optimization implementation to understand the in-order requirement. Check if there are existing tests for data loading or window performance to verify changes.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
spark
Bereich
databases, performance
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.