4paradigm / 4paradigm/OpenMLDB
Ignore the error message for offline jobs which try to register system tables without offline data
- Ngôn ngữ chính
- C++
- Star
- 1.7k
- Fork
- 331
- Merge trung bình
- 12 ngày 12 giờ
- Pull request đã merge (30 ngày)
- 1
Mô tả
**Bug Description**
Now we may get the error message for all offline jobs which has created feature platform system tables.
```
2023-11-28 16:33:49,887 INFO api.OpenmldbSession: Register empty dataframe of SYSTEM_FEATURE_PLATFORM.feature_views with schema StructType(StructField(name,StringType,true), StructField(db,StringType,true), StructField(sql,StringType,true), StructField(description,StringType,true), StructField(feature_names,StringType,true))
2023-11-28 16:33:49,900 WARN api.OpenmldbSession: Fail to register table SYSTEM_FEATURE_PLATFORM.feature_views java.lang.NullPointerException
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3307)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.registerTable(OpenmldbSession.scala:238)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$2(OpenmldbSession.scala:322)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$2$adapted(OpenmldbSession.scala:284)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$1(OpenmldbSession.scala:284)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.registerOpenmldbOfflineTable(OpenmldbSession.scala:282)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.(OpenmldbSession.scala:76)
at org.apache.spark.sql.SparkSession.(SparkSession.scala:179)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:987)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow$.runBatchSql(RunBatchAndShow.scala:31)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow$.main(RunBatchAndShow.scala:27)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow.main(RunBatchAndShow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
Hướng dẫn đóng góp
Hướng nghiên cứu
The error occurs in OpenmldbSession.scala at line 238, where registerTable calls Dataset.createOrReplaceTempView. Examine the registerOpenmldbOfflineTable method starting at line 282 to understand the flow for system tables without offline data. The fix likely involves checking for null or empty data before registration. Run the related batch job tests to verify the warning is suppressed correctly.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- java, scala, spark
- Lĩnh vực
- backend, databases
- Loại issue
- Lỗi
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 45/100