apache / apache/gluten

Codec: no forward progress made

Open
#10,412 5 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Sometimes the job fails with following error, and sometimes a rerun of the same job just succeeds

```
Job Aborted:Error: Job aborted.Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 1018) (100.67.244.114 executor 50): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Codec: no forward progress made
at org.apache.gluten.iterator.ClosableIterator.next(ClosableIterator.java:53)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream.liftedTree1$1(VeloxCelebornColumnarBatchSerializer.scala:197)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream.readValue(VeloxCelebornColumnarBatchSerializer.scala:196)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream$$anon$1.getNext(VeloxCelebornColumnarBatchSerializer.scala:149)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance$TaskDeserializationStream$$anon$1.hasNext(VeloxCelebornColumnarBatchSerializer.scala:160)

...


Retriable: False
Function: operator()
File: /gluten/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 601
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
# 4 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 5 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 6 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 7 _ZN6gluten24WholeStageResultIterator4nextEv
# 8 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 9 0x000000003001f930
```

The issue appears to be specific to the ZSTD codec. When I select LZ4 or disable compression, the job finishes successfully.
It also seems related to the output size. ZSTD works when the query returning a single column, but starts failing when the query is updated to return multiple columns.

### Gluten version

Gluten-1.3

### Spark version

Spark-3.2.x

### Spark configurations

spark.sql.parquet.compression.codec: zstd

let me know if I need to provide other info

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with spark.sql.parquet.compression.codec set to zstd and a multi-column query, then compare it with LZ4 or disabled compression. Inspect Velox Driver.cpp around line 601, VeloxCelebornColumnarBatchSerializer.scala around lines 149-197, and org.apache.gluten.iterator.ClosableIterator.next at line 53. Done means the multi-column ZSTD job completes without the no-forward-progress error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, scala
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.