filodb / filodb/FiloDB

JVM Errors/Java Nullpointer exceptions

Open
#154 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
240
Avg merge
7h 28m
Merged PRs (30d)
9

Description

**Branch, version, commit**

**OS and Environment**
CentOS Linux 7.3

**JVM version**
JDK 1.8

**Scala version**
Scala version 2.11

**Kafka and Cassandra versions and setup**
Cassandra version 3.2

**Spark version if used**
Spark 2.0

**Deployed mode**
(client/cluster on Spark Standalone/YARN/Mesos/EMR or default)
Standalone

**Actual (wrong) behavior**
scala> dfResult.write.format("filodb.spark").option("dataset", "flow_raw").option("row_keys", "exportMs").option("partition_keys", "protocolId").mode(SaveMode.Append).save()
[INFO] [05/10/2018 23:15:15.943] [main] [StatsDExtension(akka://kamon)] Starting the Kamon(StatsD) extension
[Stage 8:====> (1 + 11) / 12]#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f5204acf802, pid=3245, tid=0x00007f5041dca700
#
# JRE version: OpenJDK Runtime Environment (8.0_131-b12) (build 1.8.0_131-b12)
# Java VM: OpenJDK 64-Bit Server VM (25.131-b12 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 22484 C1 org.velvia.filo.ZeroCopyBinary$class.cachedHash64(Lorg/velvia/filo/ZeroCopyBinary;)J (140 bytes) @ 0x00007f5204acf802 [0x00007f5204acf760+0xa2]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /root/spark-2.0.0-bin-hadoop2.6/hs_err_pid3245.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
bin/spark-shell: line 44: 3245 Aborted (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"

scala> dfResult1.write.format("filodb.spark").option("dataset", "flow_raw").option("row_keys", "exportMs").option("partition_keys", "protocolId").mode(SaveMode.Append).save()
18/05/10 23:09:35 ERROR OneForOneStrategy:
java.lang.NullPointerException
at org.velvia.filo.ZeroCopyBinary$class.cachedHash64(ZeroCopyBinary.scala:77)
at org.velvia.filo.ZeroCopyUTF8String.cachedHash64(ZeroCopyBinary.scala:101)
at org.velvia.filo.ZeroCopyBinary$class.hashCode(ZeroCopyBinary.scala:85)
at org.velvia.filo.ZeroCopyUTF8String.hashCode(ZeroCopyBinary.scala:101)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.putIfAbsent(HashMap.java:1061)
at org.velvia.filo.vectors.DictUTF8Vector$.shouldMakeDict(DictUTF8Vector.scala:49)
at org.velvia.filo.vectors.UTF8PtrAppendable.optimizedVector(UTF8Vector.scala:304)
at org.velvia.filo.vectors.UTF8PtrAppendable.suboptimize(UTF8Vector.scala:294)
at org.velvia.filo.vectors.ObjectVector.optimize(ObjectVector.scala:59)

**Steps to reproduce**
scala> val files=Seq("file:/root/parquetfiles/fragment1523366588968000000.dat","file:/root/parquetfiles/fragment1523350809009000000.dat")
scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
scala> val parDF=sqlContext.read.parquet(files:_*)

scala> parDF1.count
res2: Long = 1168530

val dfResult = parDF.withColumn("exporterIp", inetToString(parDF("exporterIp"))).withColumn("srcIp", inetToString(parDF("srcIp"))).withColumn("dstIp", inetToString(parDF("dstIp"))).withColumn("nextHopIp", inetToString(parDF("nextHopIp"))).withColumn("bgpNextHopIp", inetToString(parDF("bgpNextHopIp"))).withColumn("appId", inetToString(parDF("appId"))).withColumn("policyQosClassificationHierarchy", inetToString(parDF("policyQosClassificationHierarchy"))).withColumn("protocolId", parDF("protocolId").cast(IntegerType)).withColumn("srcTos", parDF("srcTos").cast(IntegerType)).withColumn("dstTos", parDF("dstTos").cast(IntegerType)).withColumn("srcMask", parDF("srcMask").cast(StringType)).withColumn("dstMask", parDF("dstMask").cast(StringType)).withColumn("direction", parDF("direction").cast(StringType))

dfResult.write.format("filodb.spark").option("dataset", "flow_metric_raw_new").option("row_keys", "exportMs").option("partition_keys", "protocolId").mode(SaveMode.Append).save()

It throws either JVM errors or null pointer exceptions

**Logs**

```
some log
```

or as attached file (see below)

Unused parts of this template should be removed (including this line).

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Spark/Scala parquet reproduction and compare the two failures. Trace the stack from org.velvia.filo.ZeroCopyBinary.scala:77 through DictUTF8Vector.scala:49, UTF8Vector.scala:294/304, and ObjectVector.scala:59; inspect hs_err_pid3245.log if available. Done means the reproduction no longer produces the reported NPE or JVM crash, with regression coverage for the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.