apache / apache/gluten

[VL] An unloaded lazy vector cannot be wrapped by two different top level vectors

Open
#9,965 8 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
3d 17h
Merged PRs (30d)
86

Description

### Backend

VL (Velox)

### Bug description

After import this pr [8946]( https://github.com/apache/incubator-gluten/pull/8946) , There’s still a case that hasn’t been considered in this PR.
for example:
```
create table t1(id int, name string);
insert into t1 values(1,'wukong'),(2,'huahua'),(1,'wukong');
select * from
(
select id as boss_id,
name,
row_number() over(partition by id order by name desc) as rk
from t1
where name = 'wukong'
) t
where rk = 1
```

error:
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: An unloaded lazy vector cannot be wrapped by two different top level vectors.
Retriable: False
Expression: dictionaryValues_->markAsContainingLazyAndWrapped()
Context: Operator: FilterProject[2] 1
Function: setInternalState
File: /data/wen/github/velox/./velox/vector/DictionaryVector-inl.h
Line: 35

Spark3.5.4 velox plan
```
-- Project[4][expressions: (n4_3:INTEGER, hash_with_seed(42,"n2_4")), (n4_4:INTEGER, "n2_2"), (n4_5:VARCHAR, "n2_3"), (n4_6:INTEGER, "n2_4")] -> n4_3:INTEGER, n4_4:INTEGER, n4_5:VARCHAR, n4_6:INTEGER
-- TopNRowNumber[3][partition by (n2_4) order by (n2_3 DESC NULLS LAST) limit 1] -> n2_2:INTEGER, n2_3:VARCHAR, n2_4:INTEGER
-- Project[2][expressions: (n2_2:INTEGER, "n0_0"), (n2_3:VARCHAR, "n0_1"), (n2_4:INTEGER, "n0_0")] -> n2_2:INTEGER, n2_3:VARCHAR, n2_4:INTEGER
-- Filter[1][expression: and(isnotnull("n0_1"),equalto("n0_1","wukong"))] -> n0_0:INTEGER, n0_1:VARCHAR
-- TableScan[0][table: hive_table] -> n0_0:INTEGER, n0_1:VARCHAR

-- Project[5][expressions: (n5_4:INTEGER, "n0_0"), (n5_5:VARCHAR, "n0_1"), (n5_6:INTEGER, "rk_29")] -> n5_4:INTEGER, n5_5:VARCHAR, n5_6:INTEGER
-- Filter[4][expression: equalto("rk_29",1)] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER, rk_29:INTEGER
-- Window[3][STREAMING partition by [n0_2] order by [n0_1 DESC NULLS LAST] rk_29 := row_number() ROWS between UNBOUNDED PRECEDING and CURRENT ROW] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER, rk_29:INTEGER
-- OrderBy[2][n0_2 ASC NULLS FIRST, n0_1 DESC NULLS LAST] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER
-- TopNRowNumber[1][partition by (n0_2) order by (n0_1 DESC NULLS LAST) limit 1] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER
-- ValueStream[0][] -> n0_0:INTEGER, n0_1:VARCHAR, n0_2:INTEGER
```

### Gluten version

Gluten-1.3

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the SQL query with Gluten 1.3 and Spark 3.5.x, then inspect velox/vector/DictionaryVector-inl.h around markAsContainingLazyAndWrapped() and compare the behavior with PR 8946. Done means the query completes without the unloaded lazy vector error and the regression is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.