apache / apache/gluten

Reason: Error in ZlibDecompressionStream::Next in PagedInputStream StreamInfo (File Footer)

Open
#9,966 5 comments 2 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

```
Job aborted due to stage failure: Task 119 in stage 11389.0 failed 4 times, most recent failure: Lost task 119.3 in stage 11389.0 (TID 2266736) (dn014.hadoop.yl.cn executor 37950): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxException
Error Source: RUNTIME
Error Code: UNKNOWN
Reason: Error in ZlibDecompressionStream::Next in PagedInputStream StreamInfo (File Footer) input stream (DirectInputStream 16384 of 17605) State (1) remaining length (17602). error: -3 Info: File Footer
Retriable: False
Context: Split [Hive: hdfs:///user/dws_pm_wir_lte_pc_01h/p_province=GXG/p_hour=2025061201/20250612_054003_15275_cqbnd_e8b9f244-8297-4c1b-ad14-ced190bdcf43 0 - 3262913] Task Gluten_Stage_11389_TID_2266736_VTID_2101
Additional Context: Operator: TableScan[0] 0
Function: readOrSkip
File: /newdisk/addpatch-gluten/egluten/ep/build-velox/build/velox_ep/velox/dwio/common/compression/Compression.cpp
Line: 588
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox4dwio6common9exception15LoggedExceptionC2EPKcmS6_S6_RKSsS8_S8_
# 3 _ZN8facebook5velox4dwio6common11compression12_GLOBAL__N_123ZlibDecompressionStream10readOrSkipEPPKvPi.cold
# 4 _ZN6google8protobuf8internal18EpsCopyInputStream8InitFromEPNS0_2io19ZeroCopyInputStreamE
# 5 _ZN6google8protobuf8internal13MergeFromImplILb0EEEbPNS0_2io19ZeroCopyInputStreamEPNS0_11MessageLiteENS6_10ParseFlagsE
# 6 _ZN8facebook5velox4dwrf10ReaderBaseC1ERNS0_6memory10MemoryPoolESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS9_EESt10shared_ptrINS8_10encryption16DecrypterFactoryEEmmNS8_10FileFormatEbSD_INS0_6random17RandomSkipTrackerEESD_INS0_6common8ScanSpecEE
# 7 _ZN8facebook5velox4dwrf10DwrfReaderC1ERKNS0_4dwio6common13ReaderOptionsESt10unique_ptrINS4_13BufferedInputESt14default_deleteIS9_EE
# 8 _ZN8facebook5velox4dwrf10DwrfReader6createESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 9 _ZN8facebook5velox4dwrf16OrcReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 10 _ZN8facebook5velox9connector4hive11SplitReader12createReaderESt10shared_ptrINS0_6common14MetadataFilterEERKS4_INS2_16HiveColumnHandleEE
# 11 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsERKS4_INS2_16HiveColumnHandleEE
# 12 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 13 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 14 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 15 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 16 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 17 _ZN6gluten24WholeStageResultIterator4nextEv
# 18 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 19 0x00007eff29599f68

at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.GeneralInIterator.hasNext(GeneralInIterator.java:31)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:121)
at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:231)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:139)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1533)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxException
Error Source: RUNTIME
Error Code: UNKNOWN

### Gluten version

Gluten-1.2

### Spark version

Spark-3.4.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start at the ZlibDecompressionStream failure reported in Compression.cpp line 588 and trace how PagedInputStream reads the DWRF file footer during TableScan. Determine whether the failing Hive split reflects corrupted input or a decompression defect; done should include a verified explanation and resolution for the reported Spark task failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.