Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Our Spark push flow is failing with a table using var length dictionary, but the hadoop job succeeded on the same table. The problem
```
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - Caused by: java.nio.BufferOverflowException
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at java.nio.ByteBuffer.put(ByteBuffer.java:859)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putBytes(VarByteChunkSVForwardIndexWriter.java:101)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putString(VarByteChunkSVForwardIndexWriter.java:92)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator.putString(SingleValueVarByteRawIndexCreator.java:109)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:377)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:244)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2094)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2094)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.scheduler.Task.run(Task.scala:109)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:429)
```
The problem can be traced back to at least Commit 21632dadb8cd2d8b77aec523a758d73
Contributor guide
Research direction
Start at VarByteChunkSVForwardIndexWriter.putBytes (line 101) and follow calls through putString, SingleValueVarByteRawIndexCreator.putString, and SegmentColumnarIndexCreator.indexRow. Reproduce the Spark push flow with a var-length dictionary table and inspect how the buffer size is determined. Done means the affected push completes without BufferOverflowException, with behavior validated for this path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100