[Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Hi, we're using your `spark-authz` lib which started crashing for some workflows after upgrading 1.6.1 -> 1.7.1
This only happens when joining a table that contains masked columns even though they're not used in the join condition.
```
Exception in thread "main" java.lang.ClassCastException: class org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingStage1Marker cannot be cast to class org.apache.spark.sql.catalyst.plans.logical.Join (org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingStage1Marker and org.apache.spark.sql.catalyst.plans.logical.Join are in unnamed module of loader 'app')
at org.apache.spark.sql.Dataset.resolveSelfJoinCondition(Dataset.scala:1069)
at org.apache.spark.sql.Dataset.join(Dataset.scala:1117)
[..] our code
```
This should be enough to reproduce it:
```
val df0 = sqlContext.table("table_with_masked_col") // only col2 is masked which is not used
.select($"col0", $"col1")
df0.as("a")
.join(
right = df0.as("b"),
joinExprs = $"a.col0" === $"b.col0" && $"a.col1" === $"b.col1",
joinType = "left_outer") // crashes
```
We had to downgrade to 1.6.1 where this doesn't happen.
### Affects Version(s)
1.7.1
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start by reproducing the join shown in the issue with spark-authz 1.7.1 and compare it with 1.6.1. Trace the interaction between Spark's Dataset.resolveSelfJoinCondition and DataMaskingStage1Marker when masked columns are present but unused in the join. Done means the supplied self-join no longer fails with ClassCastException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- authorization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100