apache / apache/gluten

[VL] Delta Stats getting wrongly populated after merge

Open
#9,945 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

[Expected behavior]
I am running a merge job with gluten and without gluten and following are the merge statistics on delta log respectively

**With Gluten:**
```
numTargetRowsCopied: "1056"
numTargetRowsDeleted: "0"
numTargetFilesAdded: "1"
numTargetBytesAdded: "3954"
numTargetBytesRemoved: "3215"
numTargetDeletionVectorsAdded: "0"
numTargetRowsMatchedUpdated: "528"
executionTimeMs: "158406"
numTargetRowsInserted: "528"
numTargetRowsMatchedDeleted: "0"
numTargetDeletionVectorsUpdated: "0"
scanTimeMs: "4326"
numTargetRowsUpdated: "528"
numOutputRows: "2112"
numTargetDeletionVectorsRemoved: "0"
numTargetRowsNotMatchedBySourceUpdated: "0"
numTargetChangeFilesAdded: "0"
numSourceRows: "528"
numTargetFilesRemoved: "1"
numTargetRowsNotMatchedBySourceDeleted: "0"
rewriteTimeMs: "1489"`
```

**Without Gluten**
```
numTargetRowsCopied: "0"
numTargetRowsDeleted: "0"
numTargetFilesAdded: "1"
numTargetBytesAdded: "3215"
numTargetBytesRemoved: "3225"
numTargetDeletionVectorsAdded: "0"
numTargetRowsMatchedUpdated: "528"
executionTimeMs: "132701"
materializeSourceTimeMs: "114262"
numTargetRowsInserted: "0"
numTargetRowsMatchedDeleted: "0"
numTargetDeletionVectorsUpdated: "0"
scanTimeMs: "16822"
numTargetRowsUpdated: "528"
numOutputRows: "528"
numTargetDeletionVectorsRemoved: "0"
numTargetRowsNotMatchedBySourceUpdated: "0"
numTargetChangeFilesAdded: "0"
numSourceRows: "528"
numTargetFilesRemoved: "1"
numTargetRowsNotMatchedBySourceDeleted: "0"
rewriteTimeMs: "1593"
```
Where the second is actually correct. There is no change in the underlying data (both source and destination table). I have tested this for multiple jobs where this is the case.

### Gluten version

main branch

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

Gluten Version: 1.5.0-SNAPSHOT
Commit: c38c431951b3597aa203838bd7d0592974c15ceb
CMake Version: 3.28.3
System: Linux-6.8.0-1030-gcp
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU @ 2.80GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib/python3.10/dist-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

The issue names no source files or tests; start by reproducing the Delta merge with and without Gluten on Spark 3.5.x and compare the emitted Delta-log statistics. Done means the Gluten path reports the same unchanged-data counts as the non-Gluten path, with a regression test covering the case.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.