[ SDK PERFORMANCE] The execution of the show tables command takes a long time.
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 694
- PR merge metrics
- No merged PRs in 30d
Description
As shown in the above figure, CarbonShowTablesCommand obtains metadata from metastore for each table. Currently, when there are 180,000 tables, it takes a long time (about 1 hours) to run the show tables command in spark-sql shell, which needs to be optimized.
When the filter function is not invoked, it takes about 12 seconds to obtain 180,000 tables by running the show tables command.As shown in the following figure.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating CarbonShowTablesCommand and tracing how it retrieves metastore metadata for each table when running SHOW TABLES in the spark-sql shell. Compare the filtered and unfiltered paths using a catalog with roughly 180,000 tables. Done means the command returns the expected tables without taking about an hour.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala, spark
- Domain
- data-engineering, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100