4paradigm / 4paradigm/OpenMLDB

Ignore the error message for offline jobs which try to register system tables without offline data

Aperta
#3,632 0 commenti 0 reazioni 1 assegnatario Rivendicata da @tobegit3hub Vedi su GitHub
batch-engine bug
Lingua principale
C++
Stelle
1.7k
Fork
331
Merge medio
12g 12h
PR unite (30g)
1

Descrizione

**Bug Description**

Now we may get the error message for all offline jobs which has created feature platform system tables.

```
2023-11-28 16:33:49,887 INFO api.OpenmldbSession: Register empty dataframe of SYSTEM_FEATURE_PLATFORM.feature_views with schema StructType(StructField(name,StringType,true), StructField(db,StringType,true), StructField(sql,StringType,true), StructField(description,StringType,true), StructField(feature_names,StringType,true))
2023-11-28 16:33:49,900 WARN api.OpenmldbSession: Fail to register table SYSTEM_FEATURE_PLATFORM.feature_views java.lang.NullPointerException
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3307)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.registerTable(OpenmldbSession.scala:238)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$2(OpenmldbSession.scala:322)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$2$adapted(OpenmldbSession.scala:284)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.$anonfun$registerOpenmldbOfflineTable$1(OpenmldbSession.scala:284)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.registerOpenmldbOfflineTable(OpenmldbSession.scala:282)
at com._4paradigm.openmldb.batch.api.OpenmldbSession.(OpenmldbSession.scala:76)
at org.apache.spark.sql.SparkSession.(SparkSession.scala:179)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:987)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow$.runBatchSql(RunBatchAndShow.scala:31)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow$.main(RunBatchAndShow.scala:27)
at com._4paradigm.openmldb.batchjob.RunBatchAndShow.main(RunBatchAndShow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

The error occurs in OpenmldbSession.scala at line 238, where registerTable calls Dataset.createOrReplaceTempView. Examine the registerOpenmldbOfflineTable method starting at line 282 to understand the flow for system tables without offline data. The fix likely involves checking for null or empty data before registration. Run the related batch job tests to verify the warning is suppressed correctly.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java, scala, spark
Ambito
backend, databases
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.