[VL] Some corner case for unsupported column of scan
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Here are some corner cases found during the work for https://github.com/oap-project/gluten/issues/2618 .
1. "$path", "$bucket", I think this is limit by Velox: https://github.com/facebookincubator/velox/blob/3d8a8812bdd9b4f5fb6b2ca81253e023ddfaef0f/velox/connectors/hive/SplitReader.cpp#L213
```
Seq((1, 2), (3, 4), (5, 6)).toDF("$path", "$bucket").write.parquet("/tmp/veloxtest5")
spark.read.format("parquet").load("/tmp/veloxtest5").collect.toSeq
```
```
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Got type VARCHAR for field `n0_0` at position 0, but expected INTEGER.
Retriable: False
Expression: child->type()->kindEquals(type->childAt(i))
Context: Split [Hive: file:///tmp/veloxtest5/part-00000-c3312fd5-bfb0-4e78-85ac-998be024af53-c000.snappy.parquet 0 - 681] Task Gluten stage-2 task-5
Top-Level Context: Same as context.
Function: RowVector
File: /var/git/Velox/velox/vector/ComplexVector.h
Line: 71
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxException5State4makeIZNS1_C4EPKcmS5_St17basic_string_viewIcSt11char_traitsIcEES9_S9_S9_bNS1_4TypeES9_EUlRT_E_EESt10shared_ptrIKS2_ESA_SB_
# 2 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 3 _ZN8facebook5velox17VeloxRuntimeErrorC2EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bS7_
# 4 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 5 _ZN8facebook5velox9RowVectorC2EPNS0_6memory10MemoryPoolESt10shared_ptrIKNS0_4TypeEEN5boost13intrusive_ptrINS0_6BufferEEEmSt6vectorIS5_INS0_10BaseVectorEESaISF_EESt8optionalIiE
# 6 _ZN9__gnu_cxx13new_allocatorIN8facebook5velox9RowVectorEE9constructIS3_JRPNS2_6memory10MemoryPoolERKSt10shared_ptrIKNS2_7RowTypeEEN5boost13intrusive_ptrINS2_6BufferEEERiRSt6vectorISA_INS2_10BaseVectorEESaISN_EEEEEvPT_DpOT0_
# 7 _ZNSt16allocator_traitsISaIN8facebook5velox9RowVectorEEE9constructIS2_JRPNS1_6memory10MemoryPoolERKSt10shared_ptrIKNS1_7RowTypeEEN5boost13intrusive_ptrINS1_6BufferEEERiRSt6vectorISA_INS1_10BaseVectorEESaISN_EEEEEvRS3_PT_DpOT0_
```
2. row_index metadata column, now we should fall back for metadata columns, but row_index is not covered.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.