apache / apache/gluten

[VL] Iceberg metric NumDeletes

Open
#10,021 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Iceberg metric numSplits and numDeletes gets from ParquetReader and update the metric when creating DatasourceRDD, the logic is different in Gluten.
Create new metric numSplit to report the splits number.
But numDeletes should get from Velox metric which does not report now.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start at DatasourceRDD creation and compare how ParquetReader supplies numSplits and numDeletes. Then inspect the Velox metric reporting path for delete counts; the work is done when the split count and delete count are both reported consistently for Iceberg data sources.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.