[VL] An unloaded lazy vector cannot be wrapped by two different top level vectors
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 86
Description
### Backend
VL (Velox)
### Bug description
After import this pr [8946]( https://github.com/apache/incubator-gluten/pull/8946) , There’s still a case that hasn’t been considered in this PR.
for example:
```
create table t1(id int, name string);
insert into t1 values(1,'wukong'),(2,'huahua'),(1,'wukong');
select * from
(
select id as boss_id,
name,
row_number() over(partition by id order by name desc) as rk
from t1
where name = 'wukong'
) t
where rk = 1
```
error:
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: An unloaded lazy vector cannot be wrapped by two different top level vectors.
Retriable: False
Expression: dictionaryValues_->markAsContainingLazyAndWrapped()
Context: Operator: FilterProject[2] 1
Function: setInternalState
File: /data/wen/github/velox/./velox/vector/DictionaryVector-inl.h
Line: 35
Spark3.5.4 velox plan
```
-- Project[4][expressions: (n4_3:INTEGER, hash_with_seed(42,"n2_4")), (n4_4:INTEGER, "n2_2"), (n4_5:VARCHAR, "n2_3"), (n4_6:INTEGER, "n2_4")] -> n4_3:INTEGER, n4_4:INTEGER, n4_5:VARCHAR, n4_6:INTEGER
-- TopNRowNumber[3][partition by (n2_4) order by (n2_3 DESC NULLS LAST) limit 1] -> n2_2:INTEGER, n2_3:VARCHAR, n2_4:INTEGER
-- Project[2][expressions: (n2_2:INTEGER, "n0_0"), (n2_3:VARCHAR, "n0_1"), (n2_4:INTEGER, "n0_0")] -> n2_2:INTEGER, n2_3:VARCHAR, n2_4:INTEGER
-- Filter[1][expression: and(isnotnull("n0_1"),equalto("n0_1","wukong"))] -> n0_0:INTEGER, n0_1:VARCHAR
-- TableScan[0][table: hive_table] -> n0_0:INTEGER, n0_1:VARCHAR
-- Project[5][expressions: (n5_4:INTEGER, "n0_0"), (n5_5:VARCHAR, "n0_1"), (n5_6:INTEGER, "rk_29")] -> n5_4:INTEGER, n5_5:VARCHAR, n5_6:INTEGER
-- Filter[4][expression: equalto("rk_29",1)] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER, rk_29:INTEGER
-- Window[3][STREAMING partition by [n0_2] order by [n0_1 DESC NULLS LAST] rk_29 := row_number() ROWS between UNBOUNDED PRECEDING and CURRENT ROW] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER, rk_29:INTEGER
-- OrderBy[2][n0_2 ASC NULLS FIRST, n0_1 DESC NULLS LAST] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER
-- TopNRowNumber[1][partition by (n0_2) order by (n0_1 DESC NULLS LAST) limit 1] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER
-- ValueStream[0][] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER
```
### Gluten version
Gluten-1.3
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by reproducing the SQL query with Gluten 1.3 and Spark 3.5.x, then inspect velox/vector/DictionaryVector-inl.h around markAsContainingLazyAndWrapped() and compare the behavior with PR 8946. Done means the query completes without the unloaded lazy vector error and the regression is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100