4paradigm / 4paradigm/OpenMLDB

Ignore the error message for offline jobs which try to register system tables without offline data

Abierto
#3,632 0 comentarios 0 reacciones 1 asignado Reclamado por @tobegit3hub Ver en GitHub
batch-engine bug
Lenguaje dominante
C++
Estrellas
1.7k
Forks
331
Merge medio
12 d 12 h
PR fusionados (30 d)
1

Descripción

**Bug Description**

Now we may get the error message for all offline jobs which has created feature platform system tables.

```
2023-11-28 16:33:49,887 INFO api.OpenmldbSession: Register empty dataframe of SYSTEM_FEATURE_PLATFORM.feature_views with schema StructType(StructField(name,StringType,true), StructField(db,StringType,true), StructField(sql,StringType,true), StructField(description,StringType,true), StructField(feature_names,StringType,true))
2023-11-28 16:33:49,900 WARN api.OpenmldbSession: Fail to register table SYSTEM_FEATURE_PLATFORM.feature_views java.lang.NullPointerException
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3307)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.registerTable(OpenmldbSession.scala:238)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$2(OpenmldbSession.scala:322)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$2$adapted(OpenmldbSession.scala:284)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$1(OpenmldbSession.scala:284)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.registerOpenmldbOfflineTable(OpenmldbSession.scala:282)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.(OpenmldbSession.scala:76)
at org.apache.spark.sql.SparkSession.(SparkSession.scala:179)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:987)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow$.runBatchSql(RunBatchAndShow.scala:31)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow$.main(RunBatchAndShow.scala:27)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow.main(RunBatchAndShow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

The error occurs in OpenmldbSession.scala at line 238, where registerTable calls Dataset.createOrReplaceTempView. Examine the registerOpenmldbOfflineTable method starting at line 282 to understand the flow for system tables without offline data. The fix likely involves checking for null or empty data before registration. Run the related batch job tests to verify the warning is suppressed correctly.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java, scala, spark
Área
backend, databases
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.