apache / apache/gluten

[VL] Different parquet data encoding method cause bit_and/bit_or/bit_xor functions failed on arm

Open
#8,896 4 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

While I'm running [VeloxAggregateFunctionsSuite](https://github.com/apache/incubator-gluten/blob/ca2ab6ad7d9c461b7ca1eb7b032f460ce4d567ca/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxAggregateFunctionsSuite.scala#L320), I got the following error:
```
- stddev_pop
- var_samp
- var_pop
- bit_and bit_or bit_xor *** FAILED ***
Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
... ignore some output
```
you can find the full log [here](https://github.com/kevinw66/incubator-gluten/actions/runs/13649831756/job/38156232543?pr=3)

and the parquet data encoding is:
```python
>>> import pyarrow.parquet as pq
>>> meta = pq.read_metadata("/opt/gluten/backends-velox/src/test/resources/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet")
>>> meta.row_group(0).column(3).path_in_schema
'l_linenumber'
>>> meta.row_group(0).column(3).encodings
('PLAIN_DICTIONARY', 'BIT_PACKED', 'RLE')
```

After I change the encoding to `RLE_DICTIONARY`
```python
>>> import pyarrow.parquet as pq
>>> meta = pq.read_metadata("/opt/gluten/backends-velox/src/test/resources/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet")
>>> meta.row_group(0).column(3).path_in_schema
'l_linenumber'
>>> meta.row_group(0).column(3).encodings
('PLAIN', 'RLE', 'RLE_DICTIONARY')
```

The test passes:
```
- stddev_pop
- var_samp
- var_pop
- bit_and bit_or bit_xor
- corr covar_pop covar_samp
- first
- last
- ... ignore some output
```

Reference: https://parquet.apache.org/docs/file-format/data-pages/encodings/#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure in VeloxAggregateFunctionsSuite.scala around the aggregate-function tests, using the parquet resource under backends-velox/src/test/resources/tpch-data-parquet/lineitem. Compare behavior on ARM with the file's PLAIN_DICTIONARY and RLE_DICTIONARY encodings, then verify that bit_and, bit_or, and bit_xor pass with the original data encoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.