AbsaOSS / AbsaOSS/spline-spark-agent

Spline agent require more RAM for Spark application

Abierto
#914 8 comentarios 1 reacción 1 asignado Reclamado por @wajda Ver en GitHub
bug
Lenguaje dominante
Scala
Estrellas
210
Forks
102
Merge medio
1 d 1 h
PR fusionados (30 d)
1

Descripción

Hi everybody!
We faced specific attitute running Spark app with Spline agent - it needs twice more RAM or twice less CPU per executor to build lineage. If we disable spline agent our app runs on 40 executors (5 CPU each) and 64 GB ram each. But if we enable spline agent our app fails with OOM on these settings. Does anybody know how to cure it?
Here is short log of failed stage:

`Failure Reason:`
```
Job aborted due to stage failure: Task 55 in stage 20.0 failed 4 times, most recent failure: Lost task 55.3 in stage 20.0 (TID 714) (10.202.72.205 executor 89): ExecutorLostFailure (executor 89 exited caused by one of the running tasks) Reason:
The executor with id 89 exited with exit code 137(SIGKILL, possible container OOM).
```

`[rdd at :0]`
`+details:`
```
org.apache.spark.sql.Dataset.rdd(Dataset.scala:3849)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.base/java.lang.reflect.Method.invoke(Unknown Source)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror0.jinvokeraw(JavaMirrors.scala:404)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaMethodMirror.jinvoke(JavaMirrors.scala:380)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror.apply(JavaMirrors.scala:396)
za.co.absa.spline.commons.reflect.ValueExtractor.$anonfun$scalaReflectClass$1(ValueExtractor.scala:77)
scala.util.Try$.apply(Try.scala:213)
za.co.absa.spline.commons.reflect.ValueExtractor.scalaReflectClass(ValueExtractor.scala:65)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClass(ValueExtractor.scala:58)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClassHierarchy(ValueExtractor.scala:48)
za.co.absa.spline.commons.reflect.ValueExtractor.extract(ValueExtractor.scala:33)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:114)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:120)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$dump$default$2$1(ObjectStructureDumper.scala:37)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$objectToStringRec$2(ObjectStructureDumper.scala:97)
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.