apache / apache/kyuubi

[Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables

Open
#5,092 4 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

Hi, we're using your `spark-authz` lib which started crashing for some workflows after upgrading 1.6.1 -> 1.7.1
This only happens when joining a table that contains masked columns even though they're not used in the join condition.

```
Exception in thread "main" java.lang.ClassCastException: class org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingStage1Marker cannot be cast to class org.apache.spark.sql.catalyst.plans.logical.Join (org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingStage1Marker and org.apache.spark.sql.catalyst.plans.logical.Join are in unnamed module of loader 'app')
at org.apache.spark.sql.Dataset.resolveSelfJoinCondition(Dataset.scala:1069)
at org.apache.spark.sql.Dataset.join(Dataset.scala:1117)
[..] our code
```

This should be enough to reproduce it:
```
val df0 = sqlContext.table("table_with_masked_col") // only col2 is masked which is not used
.select($"col0", $"col1")

df0.as("a")
.join(
right = df0.as("b"),
joinExprs = $"a.col0" === $"b.col0" && $"a.col1" === $"b.col1",
joinType = "left_outer") // crashes
```

We had to downgrade to 1.6.1 where this doesn't happen.

### Affects Version(s)

1.7.1

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the join shown in the issue with spark-authz 1.7.1 and compare it with 1.6.1. Trace the interaction between Spark's Dataset.resolveSelfJoinCondition and DataMaskingStage1Marker when masked columns are present but unused in the join. Done means the supplied self-join no longer fails with ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
authorization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.