[VL] Iceberg metric NumDeletes
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Iceberg metric numSplits and numDeletes gets from ParquetReader and update the metric when creating DatasourceRDD, the logic is different in Gluten.
Create new metric numSplit to report the splits number.
But numDeletes should get from Velox metric which does not report now.
### Gluten version
None
Contributor guide
Research direction
Start at DatasourceRDD creation and compare how ParquetReader supplies numSplits and numDeletes. Then inspect the Velox metric reporting path for delete counts; the work is done when the split count and delete count are both reported consistently for Iceberg data sources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100