[Improvement] replace Spark original CatalogManager with GravitinoCatalogManager
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### What would you like to be improved?
Spark original CatalogManager supports register catalog with configs, doesn't support add new catalog, remove catalog, and refresh catalogs, we'd better to replace it with GravitinoCatalogManager , but where's the point to replace it?
one solution is to add a customer SQL parser, but it couldn't handle dataframe API.
the custome anaylizer rule is not practiable because it is executed after resolveCatalogs and resolveRelations rules.
### How should we improve?
_No response_
Contributor guide
Research direction
Start by locating the integration point where GravitinoCatalogManager could replace Spark's CatalogManager, then examine the SQL parser and analyzer-rule constraints described in the issue. Done means catalog registration, addition, removal, and refresh work for both SQL and DataFrame APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100