Auto-lock Scala version in Spark profiles
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 22
Description
## Background
Currently, when developing in IntelliJ IDEA, users need to manually select both a Spark profile (e.g., `spark-3.5`) and a Scala profile (e.g., `scala-2.12`) to ensure correct compilation. This two-step configuration is error-prone and can lead to compatibility issues if mismatched.
## Proposed Change
Automatically lock the Scala version and compiler configuration when a Spark profile is selected, eliminating the need to manually select a separate Scala profile.
### Mapping
| Spark Profile | Auto-locked Scala Version | Compiler Plugins |
|---------------|---------------------------|------------------|
| spark-3.0 ~ 3.5 | 2.12.18 | semanticdb + paradise |
| spark-4.0 ~ 4.1 | 2.13.17 | semanticdb only (with -Ymacro-annotations) |
## Implementation Details
For each Spark profile, add:
1. **Properties**:
```xml
2.12
2.12.18
```
2. **scala-maven-plugin configuration**:
- **Spark 3.x**: Add `semanticdb-scalac` + `paradise` compiler plugins
- **Spark 4.x**: Add `-Ymacro-annotations` args + `semanticdb-scalac` plugin (no paradise needed)
- Use `combine.self="override"` to ensure profile configurations don't merge with base configurations
## Benefits
- **Simplified workflow**: Only select one profile (e.g., `spark-3.5`) instead of two
- **Reduced errors**: Eliminates Scala/Spark version mismatch issues
- **Better DX**: Clearer intent and easier onboarding for new developers
## Testing
Verified in IntelliJ IDEA by:
1. Selecting only `spark-3.5` profile → Scala 2.12.18 correctly configured
2. Selecting only `spark-4.0` profile → Scala 2.13.17 correctly configured
3. No need to select separate `scala-2.12` or `scala-2.13` profiles
Contributor guide
Assessment
This issue has not been assessed yet.