4paradigm / 4paradigm/OpenMLDB

Support UnsafeRowOpt for groupby agg physical node

Offen
#1,346 1 Kommentar 0 Reaktionen 1 zugewiesene Person Beansprucht von @tobegit3hub Auf GitHub ansehen
enhancement
Vorherrschende Sprache
C++
Sterne
1.7k
Forks
331
Ø Merge
12 T. 12 Std.
Gemergte PRs (30 T.)
1

Beschreibung

Now we can not enable UnsafeRowOpt for SQL with `group by` which may output incorrect result or crash because of C++ core.

Here is the simple case to reproduce.

```
test("Test unsafe groupby") {
val spark = getSparkSession
val sess = new OpenmldbSession(spark)

val data = Seq(
Row(1, "tom", 100, 1),
Row(2, "amy", 200, 2),
Row(3, "tom", 300, 3),
Row(4, "amy", 400, 4),
Row(5, "tom", 500, 5),
Row(6, "amy", 600, 6),
Row(7, "tom", 700, 7),
Row(8, "amy", 800, 8),
Row(9, "tom", 900, 9),
Row(10, "amy", 1000, 10))
val schema = StructType(List(
StructField("id", IntegerType),
StructField("user", StringType),
StructField("trans_amount", IntegerType),
StructField("trans_time", IntegerType)))
val df = spark.createDataFrame(spark.sparkContext.makeRDD(data), schema)

sess.registerTable("t1", df)
df.createOrReplaceTempView("t1")

val sqlText = "SELECT max(id) AS max_id, sum(trans_amount) AS sum_amount FROM t1 GROUP BY user"
// core for unaligned memory
//val sqlText = "SELECT user, max(id) AS max_id, sum(trans_amount) AS sum_amount FROM t1 GROUP BY user"

val outputDf = sess.sql(sqlText)
outputDf.show()
}
```

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue involves the groupby aggregation physical node in OpenMLDB's execution engine, specifically the UnsafeRowOpt optimization. Start by examining the code around groupby physical node implementation and UnsafeRowOpt handling. Look for memory alignment issues or incorrect result logic when UnsafeRowOpt is enabled. The provided test case reproduces the crash; run it to see the failure, then trace through the relevant C++ code to understand the root cause.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
spark, sql
Bereich
backend, databases, machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.