4paradigm / 4paradigm/OpenMLDB
Support UnsafeRowOpt for groupby agg physical node
- 主要语言
- C++
- 星标
- 1.7k
- 派生
- 331
- 平均合并
- 12 天 12 小时
- 30 天内合并 PR
- 1
描述
Now we can not enable UnsafeRowOpt for SQL with `group by` which may output incorrect result or crash because of C++ core.
Here is the simple case to reproduce.
```
test("Test unsafe groupby") {
val spark = getSparkSession
val sess = new OpenmldbSession(spark)
val data = Seq(
Row(1, "tom", 100, 1),
Row(2, "amy", 200, 2),
Row(3, "tom", 300, 3),
Row(4, "amy", 400, 4),
Row(5, "tom", 500, 5),
Row(6, "amy", 600, 6),
Row(7, "tom", 700, 7),
Row(8, "amy", 800, 8),
Row(9, "tom", 900, 9),
Row(10, "amy", 1000, 10))
val schema = StructType(List(
StructField("id", IntegerType),
StructField("user", StringType),
StructField("trans_amount", IntegerType),
StructField("trans_time", IntegerType)))
val df = spark.createDataFrame(spark.sparkContext.makeRDD(data), schema)
sess.registerTable("t1", df)
df.createOrReplaceTempView("t1")
val sqlText = "SELECT max(id) AS max_id, sum(trans_amount) AS sum_amount FROM t1 GROUP BY user"
// core for unaligned memory
//val sqlText = "SELECT user, max(id) AS max_id, sum(trans_amount) AS sum_amount FROM t1 GROUP BY user"
val outputDf = sess.sql(sqlText)
outputDf.show()
}
```
贡献指南
调研方向
The issue involves the groupby aggregation physical node in OpenMLDB's execution engine, specifically the UnsafeRowOpt optimization. Start by examining the code around groupby physical node implementation and UnsafeRowOpt handling. Look for memory alignment issues or incorrect result logic when UnsafeRowOpt is enabled. The provided test case reproduces the crash; run it to see the failure, then trace through the relevant C++ code to understand the root cause.
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- spark, sql
- 领域
- backend, databases, machine-learning
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100