[Performance] Avoid N+1 remote table enumeration in external session bypass SHOW TABLES path
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Summary
In the external session-bypass path, `ExternalDatabase.getTables()` performs **N+1 remote table-name enumerations**.
When `extCatalog.shouldBypassTableNameCache(sessionContext)` is true:
1. `getTables()` calls `getTableNamesWithLock()`, which enumerates remote table names once via `listLocalTableNamesWithoutCache(sessionContext)`.
2. Then `getTables()` iterates all table names and calls `getTableNullable(tblName)`.
3. In bypass mode, `getTableNullable(tblName)` enters `getTableNullableWithoutCache(sessionContext, tableName)`.
4. `getTableNullableWithoutCache(...)` calls `findTableNamePairWithoutCache(...)`, which again does a full `listTableNames(sessionContext)` scan for each table.
As a result, listing `N` tables triggers `1 + N` remote `LIST TABLES` enumerations.
## Affected code
`fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java`
Current structure:
```java
public List getTables() {
Set tblNames = getTableNamesWithLock(); // 1 full remote enumeration in bypass mode
for (String tblName : tblNames) {
T tbl = getTableNullable(tblName); // each iteration re-enters bypass path
}
}
public Set getTableNamesWithLock() {
if (extCatalog.shouldBypassTableNameCache(sessionContext)) {
return Sets.newHashSet(listLocalTableNamesWithoutCache(sessionContext));
}
}
private T getTableNullableWithoutCache(SessionContext sessionContext, String tableName) {
Optional> matched = findTableNamePairWithoutCache(sessionContext, tableName);
// findTableNamePairWithoutCache() -> listTableNames(sessionContext)
}
```
## Impact
This affects session-bypass catalogs such as delegated Iceberg REST session catalogs.
For `SHOW TABLES` / `getTables()` on a database with many tables, the current implementation scales poorly because every table object construction repeats a full remote name enumeration.
The issue is performance-only, but the overhead can become significant for large databases or expensive remote catalogs.
## Scope / origin
This is a **pre-existing master/baseline issue**, not introduced by PR #65126.
The bypass structure already existed before the external meta cache refactor. PR #65126 carried it forward and added related session-bypass improvements elsewhere, but did not address this `getTables()` path.
## Suggested fix
In bypass mode, reuse a **single remote name enumeration** inside `getTables()`:
- enumerate `(remoteName, localName)` pairs once,
- build a local lookup map from the result,
- construct table objects directly from that snapshot instead of re-calling `getTableNullable()` for every table.
This keeps bypass semantics unchanged while removing the repeated full scans.
## Related
- PR #65126
- Commit `48291325300 [fix](fe) Avoid unnecessary session table enumeration` addressed a nearby delegated-session existence-check path, but not this `getTables()` N+1 path.
Contributor guide
Research direction
Start in fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java at getTables(), then trace getTableNamesWithLock(), getTableNullableWithoutCache(), and findTableNamePairWithoutCache(). Verify the SHOW TABLES/getTables path for session-bypass catalogs reuses one remote name enumeration while preserving table construction and bypass behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100