apache / apache/datafusion-comet

Spark 4.2: SQL Last Attempt Metric (SLAM) not propagated through Comet operators

Open
#4,963 0 comments 0 reactions 0 assignees View on GitHub
bug priority:low spark 4.2 spark sql tests
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Spark 4.2 adds a "SQL Last Attempt Metric" (SLAM) feature and test suites that assert on the metric value recorded for a query's executed plan.

When Comet is enabled, Comet replaces the Spark operators with native equivalents and the SLAM metric value is not propagated through the plan, so `lastAttemptValueForDataset` / `lastAttemptValueForHighestRDDId` return `None`.

Affected Spark SQL test suites (currently marked `IgnoreComet` / `IgnoreCometSuite` in `dev/diffs/4.2.0.diff`):
- `SQLLastAttemptMetricPlanShapesSuite` (whole suite)
- `MetricsFailureInjectionSuite` (whole suite)
- `SQLLastAttemptMetricIntegrationSuite` ("dataset updates")

Contributor guide

Open the contributing guide

Research direction

Start with dev/diffs/4.2.0.diff and the ignored SQLLastAttemptMetricPlanShapesSuite, MetricsFailureInjectionSuite, and SQLLastAttemptMetricIntegrationSuite dataset-updates tests. Trace how Comet’s replacement operators handle the SLAM value, then restore propagation so these suites can run without IgnoreComet/IgnoreCometSuite and their metric assertions pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.