apache / apache/kyuubi

[Umbrella] [Authz] Support multiple Ranger services for different Spark catalogs

Open
#3,586 0 comments 2 reactions 0 assignees View on GitHub
kind:umbrella priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues) and found no similar issues.

### Describe the proposal

Currently, Kyuubi Authz plugin inits a singleton plugin instance that only supports single Ranger policy service for all Spark catalogs. This limits the use range of the plugin with one default catalog or it will introduce chaos when querying in different catalogs.

To solve this problem, we will introduce multiple service support for different Spark V2 catalogs.
Key points as listed,

- reserve default Ranger plugin instance as default fallback handler for all access privilege checking, row-filter and data masking for any default catalog or no explicit catalog identified.
- continuing to use Hive service def for building privilege types and operation types
- parsing catalog names from `DataSourceV2Relation` LogicalPlans in `Privilegebuilder`
- configuring and initialing RangerPlugin with the mapping of the catalog name to each Ranger service name
- support catalog in `AccessRequest`, row-filter and datamasking and diverting requests to the corresponding RangerPlugin instance

### Task list

- [ ] (abandoned)#3594
- [x] #3605
- [x] #3984
- [ ] #4015
- [ ] #4032

1. PrivilegeObject -> AccessResource 加catalog字段并传递
2. Desciptor -> PrivilegeObject 传递 catalog,Desciptor新增Database类,拓展Function类、Table类 加catalog字段
3. spec -> Desciptor 提取catlog并填入,加入catalog extractor (for dsv2), 及在tablespec和dbspec加catalog spec

### Are you willing to submit PR?

- [X] Yes. I can submit a PR independently to improve.
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the completed and open task issues, then trace the listed PrivilegeObject, Descriptor, spec, Privilegebuilder, AccessRequest, row-filter, and data-masking components. The proposal's done state is catalog information flowing through privilege checks and requests to the corresponding RangerPlugin, with the default instance as fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
authorization, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.