apache / apache/gluten

[VL] Gluten write more shuffle data than vanilla Spark

Open
#8,833 11 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

I can see that for the same record numbers, gluten always write much larger shuffle data than vanilla, why? From my understanding, gluten should use less storage since it's written in columnar format.

gluten

![Image](https://github.com/user-attachments/assets/3d74f63f-ac27-4852-a585-dc07e46a4e89)

vanilla

![Image](https://github.com/user-attachments/assets/e6d35121-3347-4a69-89a0-d9bc87072a8a)

### Spark version

Spark-3.2.x

### Spark configurations

_No response_

### System information

gluten version: 1.3.0

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.