apache / apache/gluten

count distinct and from_unixtime return different result compared to Vanilla Spark

Open
#4,518 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

When using count distinct and from_unixtime, the result shows difference with Vanilla Spark.

Below is the reproduce case:
select a1
,count(1)
, 1000/count(distinct if(a1=a2, b, null)) as send_check_avg_cnt_sd
, from_unixtime(unix_timestamp(max(d),'yyyyMMdd'),'yyyy-MM-dd') as last_wb_detail_view_day,max(d)
, count(distinct if(a1=a2 and c in ('4'), b, null)) as consignor_check_waybill
from count2
group by a1 order by a1 limit 100 ;

### Spark version

Spark-3.2.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.