Reason: Error in ZlibDecompressionStream::Next in PagedInputStream StreamInfo (File Footer)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
```
Job aborted due to stage failure: Task 119 in stage 11389.0 failed 4 times, most recent failure: Lost task 119.3 in stage 11389.0 (TID 2266736) (dn014.hadoop.yl.cn executor 37950): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxException
Error Source: RUNTIME
Error Code: UNKNOWN
Reason: Error in ZlibDecompressionStream::Next in PagedInputStream StreamInfo (File Footer) input stream (DirectInputStream 16384 of 17605) State (1) remaining length (17602). error: -3 Info: File Footer
Retriable: False
Context: Split [Hive: hdfs:///user/dws_pm_wir_lte_pc_01h/p_province=GXG/p_hour=2025061201/20250612_054003_15275_cqbnd_e8b9f244-8297-4c1b-ad14-ced190bdcf43 0 - 3262913] Task Gluten_Stage_11389_TID_2266736_VTID_2101
Additional Context: Operator: TableScan[0] 0
Function: readOrSkip
File: /newdisk/addpatch-gluten/egluten/ep/build-velox/build/velox_ep/velox/dwio/common/compression/Compression.cpp
Line: 588
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox4dwio6common9exception15LoggedExceptionC2EPKcmS6_S6_RKSsS8_S8_
# 3 _ZN8facebook5velox4dwio6common11compression12_GLOBAL__N_123ZlibDecompressionStream10readOrSkipEPPKvPi.cold
# 4 _ZN6google8protobuf8internal18EpsCopyInputStream8InitFromEPNS0_2io19ZeroCopyInputStreamE
# 5 _ZN6google8protobuf8internal13MergeFromImplILb0EEEbPNS0_2io19ZeroCopyInputStreamEPNS0_11MessageLiteENS6_10ParseFlagsE
# 6 _ZN8facebook5velox4dwrf10ReaderBaseC1ERNS0_6memory10MemoryPoolESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS9_EESt10shared_ptrINS8_10encryption16DecrypterFactoryEEmmNS8_10FileFormatEbSD_INS0_6random17RandomSkipTrackerEESD_INS0_6common8ScanSpecEE
# 7 _ZN8facebook5velox4dwrf10DwrfReaderC1ERKNS0_4dwio6common13ReaderOptionsESt10unique_ptrINS4_13BufferedInputESt14default_deleteIS9_EE
# 8 _ZN8facebook5velox4dwrf10DwrfReader6createESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 9 _ZN8facebook5velox4dwrf16OrcReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 10 _ZN8facebook5velox9connector4hive11SplitReader12createReaderESt10shared_ptrINS0_6common14MetadataFilterEERKS4_INS2_16HiveColumnHandleEE
# 11 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsERKS4_INS2_16HiveColumnHandleEE
# 12 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 13 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 14 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 15 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 16 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 17 _ZN6gluten24WholeStageResultIterator4nextEv
# 18 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 19 0x00007eff29599f68
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.GeneralInIterator.hasNext(GeneralInIterator.java:31)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:121)
at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:231)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:139)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1533)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxException
Error Source: RUNTIME
Error Code: UNKNOWN
### Gluten version
Gluten-1.2
### Spark version
Spark-3.4.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start at the ZlibDecompressionStream failure reported in Compression.cpp line 588 and trace how PagedInputStream reads the DWRF file footer during TableScan. Determine whether the failing Hive split reflects corrupted input or a decompression defect; done should include a verified explanation and resolution for the reported Spark task failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100