Codec: no forward progress made
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Sometimes the job fails with following error, and sometimes a rerun of the same job just succeeds
```
Job Aborted:Error: Job aborted.Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 1018) (100.67.244.114 executor 50): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Codec: no forward progress made
at org.apache.gluten.iterator.ClosableIterator.next(ClosableIterator.java:53)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream.liftedTree1$1(VeloxCelebornColumnarBatchSerializer.scala:197)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream.readValue(VeloxCelebornColumnarBatchSerializer.scala:196)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream$$anon$1.getNext(VeloxCelebornColumnarBatchSerializer.scala:149)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream$$anon$1.hasNext(VeloxCelebornColumnarBatchSerializer.scala:160)
...
Retriable: False
Function: operator()
File: /gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 601
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
# 4 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 5 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 6 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 7 _ZN6gluten24WholeStageResultIterator4nextEv
# 8 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 9 0x000000003001f930
```
The issue appears to be specific to the ZSTD codec. When I select LZ4 or disable compression, the job finishes successfully.
It also seems related to the output size. ZSTD works when the query returning a single column, but starts failing when the query is updated to return multiple columns.
### Gluten version
Gluten-1.3
### Spark version
Spark-3.2.x
### Spark configurations
spark.sql.parquet.compression.codec: zstd
let me know if I need to provide other info
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by reproducing the failure with spark.sql.parquet.compression.codec set to zstd and a multi-column query, then compare it with LZ4 or disabled compression. Inspect Velox Driver.cpp around line 601, VeloxCelebornColumnarBatchSerializer.scala around lines 149-197, and org.apache.gluten.iterator.ClosableIterator.next at line 53. Done means the multi-column ZSTD job completes without the no-forward-progress error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java, scala
- Domain
- backend, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100