apache / apache/gluten

[VL] IncrementMetric doesn't work in some cases

Open
#9,003 13 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

The following test prints:

metric should be 3. Value: 0
metric should be 3. Value: 3

```
test("operation metrics - test gluten") {
withTempDir { tempDir =>
val dirPath = tempDir.getAbsolutePath
spark.range(0, 3)
.write
.format("delta")
.save(dirPath)

val metric1 = createMetric(spark.sparkContext, "number of rows")
val incrMetric1 = IncrementMetric(TrueLiteral, metric1)

spark.read.load(dirPath)
.select(col("_metadata.file_path"))
.filter(new Column(incrMetric1))
.collect()

println("metric should be 3. Value: " + metric1.value)

val metric2 = createMetric(spark.sparkContext, "number of rows")
val incrMetric2 = IncrementMetric(TrueLiteral, metric2)

spark.read.load(dirPath)
.select(input_file_name())
.filter(new Column(incrMetric2))
.collect()

println("metric should be 3. Value: " + metric2.value)
}
}
```

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.