apache / apache/datafusion-comet
Spark 4.2: SQL Last Attempt Metric (SLAM) not propagated through Comet operators
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Spark 4.2 adds a "SQL Last Attempt Metric" (SLAM) feature and test suites that assert on the metric value recorded for a query's executed plan.
When Comet is enabled, Comet replaces the Spark operators with native equivalents and the SLAM metric value is not propagated through the plan, so `lastAttemptValueForDataset` / `lastAttemptValueForHighestRDDId` return `None`.
Affected Spark SQL test suites (currently marked `IgnoreComet` / `IgnoreCometSuite` in `dev/diffs/4.2.0.diff`):
- `SQLLastAttemptMetricPlanShapesSuite` (whole suite)
- `MetricsFailureInjectionSuite` (whole suite)
- `SQLLastAttemptMetricIntegrationSuite` ("dataset updates")
Contributor guide
Research direction
Start with dev/diffs/4.2.0.diff and the ignored SQLLastAttemptMetricPlanShapesSuite, MetricsFailureInjectionSuite, and SQLLastAttemptMetricIntegrationSuite dataset-updates tests. Trace how Comet’s replacement operators handle the SLAM value, then restore propagation so these suites can run without IgnoreComet/IgnoreCometSuite and their metric assertions pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100