apache / apache/doris

[Performance] Avoid N+1 remote table enumeration in external session bypass SHOW TABLES path

Open
#66,025 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Summary

In the external session-bypass path, `ExternalDatabase.getTables()` performs **N+1 remote table-name enumerations**.

When `extCatalog.shouldBypassTableNameCache(sessionContext)` is true:

1. `getTables()` calls `getTableNamesWithLock()`, which enumerates remote table names once via `listLocalTableNamesWithoutCache(sessionContext)`.
2. Then `getTables()` iterates all table names and calls `getTableNullable(tblName)`.
3. In bypass mode, `getTableNullable(tblName)` enters `getTableNullableWithoutCache(sessionContext, tableName)`.
4. `getTableNullableWithoutCache(...)` calls `findTableNamePairWithoutCache(...)`, which again does a full `listTableNames(sessionContext)` scan for each table.

As a result, listing `N` tables triggers `1 + N` remote `LIST TABLES` enumerations.

## Affected code

`fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java`

Current structure:

```java
public List getTables() {
Set tblNames = getTableNamesWithLock(); // 1 full remote enumeration in bypass mode
for (String tblName : tblNames) {
T tbl = getTableNullable(tblName); // each iteration re-enters bypass path
}
}

public Set getTableNamesWithLock() {
if (extCatalog.shouldBypassTableNameCache(sessionContext)) {
return Sets.newHashSet(listLocalTableNamesWithoutCache(sessionContext));
}
}

private T getTableNullableWithoutCache(SessionContext sessionContext, String tableName) {
Optional> matched = findTableNamePairWithoutCache(sessionContext, tableName);
// findTableNamePairWithoutCache() -> listTableNames(sessionContext)
}
```

## Impact

This affects session-bypass catalogs such as delegated Iceberg REST session catalogs.

For `SHOW TABLES` / `getTables()` on a database with many tables, the current implementation scales poorly because every table object construction repeats a full remote name enumeration.

The issue is performance-only, but the overhead can become significant for large databases or expensive remote catalogs.

## Scope / origin

This is a **pre-existing master/baseline issue**, not introduced by PR #65126.

The bypass structure already existed before the external meta cache refactor. PR #65126 carried it forward and added related session-bypass improvements elsewhere, but did not address this `getTables()` path.

## Suggested fix

In bypass mode, reuse a **single remote name enumeration** inside `getTables()`:

- enumerate `(remoteName, localName)` pairs once,
- build a local lookup map from the result,
- construct table objects directly from that snapshot instead of re-calling `getTableNullable()` for every table.

This keeps bypass semantics unchanged while removing the repeated full scans.

## Related

- PR #65126
- Commit `48291325300 [fix](fe) Avoid unnecessary session table enumeration` addressed a nearby delegated-session existence-check path, but not this `getTables()` N+1 path.

Contributor guide

Open the contributing guide

Research direction

Start in fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java at getTables(), then trace getTableNamesWithLock(), getTableNullableWithoutCache(), and findTableNamePairWithoutCache(). Verify the SHOW TABLES/getTables path for session-bypass catalogs reuses one remote name enumeration while preserving table construction and bypass behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.