[Improvement] Improve the current SQL of JDBCBackend
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### What would you like to be improved?
In the current implementation of xxxMetaMapper and xxxMetaService, we seldom use JOIN and transactions, for example:
The list catalog operation will issue two SQL queries, the first one is to get a metalake id, then using this id to get the catalog list.
Instead of issuing two queries, we can use JOIN to join catalog and metalake table to get catalog list by one query, this will save the IO time and avoid inconsistent problem.
Also for `updateCatalog`
We have several steps:
1. get metalake id.
2. get catalog PO.
3. update the catalog object.
4. update the catalog in RDBMS.
These 4 steps are not in the transaction, which will potentially meet the inconsistent problem, and we highly leverage the lock to avoid the concurrent problem. A better solution is to put these 4 steps into a transaction.
### How should we improve?
So basically, we can improve the current SQL to:
1. reduce the queries and increase the performance.
2. Put multiple step actions into one transaction to leverage DB's transaction to keep consistent.
Contributor guide
Research direction
Start in the JDBCBackend implementation, locating the xxxMetaMapper and xxxMetaService code for list catalog and updateCatalog. Trace the current metalake, catalog, and update operations and review how JDBC transactions are handled. Done means reducing redundant SQL queries with appropriate joins and grouping the multi-step update flow in a transaction without inconsistency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100