AbsaOSS / AbsaOSS/spline-spark-agent

Spline agent require more RAM for Spark application

Offen
#914 8 Kommentare 1 Reaktion 1 zugewiesene Person Beansprucht von @wajda Auf GitHub ansehen
bug
Vorherrschende Sprache
Scala
Sterne
210
Forks
102
Ø Merge
1 T. 1 Std.
Gemergte PRs (30 T.)
1

Beschreibung

Hi everybody!
We faced specific attitute running Spark app with Spline agent - it needs twice more RAM or twice less CPU per executor to build lineage. If we disable spline agent our app runs on 40 executors (5 CPU each) and 64 GB ram each. But if we enable spline agent our app fails with OOM on these settings. Does anybody know how to cure it?
Here is short log of failed stage:

`Failure Reason:`
```
Job aborted due to stage failure: Task 55 in stage 20.0 failed 4 times, most recent failure: Lost task 55.3 in stage 20.0 (TID 714) (10.202.72.205 executor 89): ExecutorLostFailure (executor 89 exited caused by one of the running tasks) Reason:
The executor with id 89 exited with exit code 137(SIGKILL, possible container OOM).
```

`[rdd at :0]`
`+details:`
```
org.apache.spark.sql.Dataset.rdd(Dataset.scala:3849)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.base/java.lang.reflect.Method.invoke(Unknown Source)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror0.jinvokeraw(JavaMirrors.scala:404)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaMethodMirror.jinvoke(JavaMirrors.scala:380)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror.apply(JavaMirrors.scala:396)
za.co.absa.spline.commons.reflect.ValueExtractor.$anonfun$scalaReflectClass$1(ValueExtractor.scala:77)
scala.util.Try$.apply(Try.scala:213)
za.co.absa.spline.commons.reflect.ValueExtractor.scalaReflectClass(ValueExtractor.scala:65)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClass(ValueExtractor.scala:58)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClassHierarchy(ValueExtractor.scala:48)
za.co.absa.spline.commons.reflect.ValueExtractor.extract(ValueExtractor.scala:33)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:114)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:120)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$dump$default$2$1(ObjectStructureDumper.scala:37)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$objectToStringRec$2(ObjectStructureDumper.scala:97)
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.