AbsaOSS / AbsaOSS/spline-spark-agent

Spline agent require more RAM for Spark application

Aperta
#914 8 commenti 1 reazione 1 assegnatario Rivendicata da @wajda Vedi su GitHub
bug
Lingua principale
Scala
Stelle
210
Fork
102
Merge medio
1g 1h
PR unite (30g)
1

Descrizione

Hi everybody!
We faced specific attitute running Spark app with Spline agent - it needs twice more RAM or twice less CPU per executor to build lineage. If we disable spline agent our app runs on 40 executors (5 CPU each) and 64 GB ram each. But if we enable spline agent our app fails with OOM on these settings. Does anybody know how to cure it?
Here is short log of failed stage:

`Failure Reason:`
```
Job aborted due to stage failure: Task 55 in stage 20.0 failed 4 times, most recent failure: Lost task 55.3 in stage 20.0 (TID 714) (10.202.72.205 executor 89): ExecutorLostFailure (executor 89 exited caused by one of the running tasks) Reason:
The executor with id 89 exited with exit code 137(SIGKILL, possible container OOM).
```

`[rdd at :0]`
`+details:`
```
org.apache.spark.sql.Dataset.rdd(Dataset.scala:3849)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.base/java.lang.reflect.Method.invoke(Unknown Source)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror0.jinvokeraw(JavaMirrors.scala:404)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaMethodMirror.jinvoke(JavaMirrors.scala:380)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror.apply(JavaMirrors.scala:396)
za.co.absa.spline.commons.reflect.ValueExtractor.$anonfun$scalaReflectClass$1(ValueExtractor.scala:77)
scala.util.Try$.apply(Try.scala:213)
za.co.absa.spline.commons.reflect.ValueExtractor.scalaReflectClass(ValueExtractor.scala:65)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClass(ValueExtractor.scala:58)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClassHierarchy(ValueExtractor.scala:48)
za.co.absa.spline.commons.reflect.ValueExtractor.extract(ValueExtractor.scala:33)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:114)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:120)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$dump$default$2$1(ObjectStructureDumper.scala:37)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$objectToStringRec$2(ObjectStructureDumper.scala:97)
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.