apache / apache/auron

Auto-lock Scala version in Spark profiles

Open
#2,013 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

## Background

Currently, when developing in IntelliJ IDEA, users need to manually select both a Spark profile (e.g., `spark-3.5`) and a Scala profile (e.g., `scala-2.12`) to ensure correct compilation. This two-step configuration is error-prone and can lead to compatibility issues if mismatched.

## Proposed Change

Automatically lock the Scala version and compiler configuration when a Spark profile is selected, eliminating the need to manually select a separate Scala profile.

### Mapping

| Spark Profile | Auto-locked Scala Version | Compiler Plugins |
|---------------|---------------------------|------------------|
| spark-3.0 ~ 3.5 | 2.12.18 | semanticdb + paradise |
| spark-4.0 ~ 4.1 | 2.13.17 | semanticdb only (with -Ymacro-annotations) |

## Implementation Details

For each Spark profile, add:

1. **Properties**:
```xml
2.12
2.12.18
```

2. **scala-maven-plugin configuration**:
- **Spark 3.x**: Add `semanticdb-scalac` + `paradise` compiler plugins
- **Spark 4.x**: Add `-Ymacro-annotations` args + `semanticdb-scalac` plugin (no paradise needed)
- Use `combine.self="override"` to ensure profile configurations don't merge with base configurations

## Benefits

- **Simplified workflow**: Only select one profile (e.g., `spark-3.5`) instead of two
- **Reduced errors**: Eliminates Scala/Spark version mismatch issues
- **Better DX**: Clearer intent and easier onboarding for new developers

## Testing

Verified in IntelliJ IDEA by:
1. Selecting only `spark-3.5` profile → Scala 2.12.18 correctly configured
2. Selecting only `spark-4.0` profile → Scala 2.13.17 correctly configured
3. No need to select separate `scala-2.12` or `scala-2.13` profiles

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.