[VL] Delta Stats getting wrongly populated after merge
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
[Expected behavior]
I am running a merge job with gluten and without gluten and following are the merge statistics on delta log respectively
**With Gluten:**
```
numTargetRowsCopied: "1056"
numTargetRowsDeleted: "0"
numTargetFilesAdded: "1"
numTargetBytesAdded: "3954"
numTargetBytesRemoved: "3215"
numTargetDeletionVectorsAdded: "0"
numTargetRowsMatchedUpdated: "528"
executionTimeMs: "158406"
numTargetRowsInserted: "528"
numTargetRowsMatchedDeleted: "0"
numTargetDeletionVectorsUpdated: "0"
scanTimeMs: "4326"
numTargetRowsUpdated: "528"
numOutputRows: "2112"
numTargetDeletionVectorsRemoved: "0"
numTargetRowsNotMatchedBySourceUpdated: "0"
numTargetChangeFilesAdded: "0"
numSourceRows: "528"
numTargetFilesRemoved: "1"
numTargetRowsNotMatchedBySourceDeleted: "0"
rewriteTimeMs: "1489"`
```
**Without Gluten**
```
numTargetRowsCopied: "0"
numTargetRowsDeleted: "0"
numTargetFilesAdded: "1"
numTargetBytesAdded: "3215"
numTargetBytesRemoved: "3225"
numTargetDeletionVectorsAdded: "0"
numTargetRowsMatchedUpdated: "528"
executionTimeMs: "132701"
materializeSourceTimeMs: "114262"
numTargetRowsInserted: "0"
numTargetRowsMatchedDeleted: "0"
numTargetDeletionVectorsUpdated: "0"
scanTimeMs: "16822"
numTargetRowsUpdated: "528"
numOutputRows: "528"
numTargetDeletionVectorsRemoved: "0"
numTargetRowsNotMatchedBySourceUpdated: "0"
numTargetChangeFilesAdded: "0"
numSourceRows: "528"
numTargetFilesRemoved: "1"
numTargetRowsNotMatchedBySourceDeleted: "0"
rewriteTimeMs: "1593"
```
Where the second is actually correct. There is no change in the underlying data (both source and destination table). I have tested this for multiple jobs where this is the case.
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
Gluten Version: 1.5.0-SNAPSHOT
Commit: c38c431951b3597aa203838bd7d0592974c15ceb
CMake Version: 3.28.3
System: Linux-6.8.0-1030-gcp
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU @ 2.80GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib/python3.10/dist-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
```
Contributor guide
Research direction
The issue names no source files or tests; start by reproducing the Delta merge with and without Gluten on Spark 3.5.x and compare the emitted Delta-log statistics. Done means the Gluten path reports the same unchanged-data counts as the non-Gluten path, with a regression test covering the case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100