matrixorigin / matrixorigin/matrixone
[Tech Request]: Tenant-Aware Catalog Logtail Subscription for CN Nodes
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Is there an existing issue for the same tech request?
- [x] I have checked the existing issues.
## Does this tech request not affect user experience?
- [x] This tech request doesn't affect user experience.
## What would you like to be added?
### Tenant-Aware Catalog Logtail Subscription for CN Nodes
Currently, when a CN starts up, it subscribes to the full logtail of system catalog tables (mo_database, mo_tables, mo_columns) via PushClient.subSysTables(). The TN sends all tenants catalog data to every CN, and the CatalogCache stores entries for all accounts in memory, only filtering by AccountId at query time.
**Current flow:**
1. CN calls subSysTables() -> subscribes to MO_DATABASE_ID, MO_TABLES_ID, MO_COLUMNS_ID (no tenant filter)
2. TN collects and pushes logtail for these tables containing all tenants data
3. CN applyToCatalogCache() inserts all rows into CatalogCache (btree of TableItem/DatabaseItem)
4. Query-time filtering: CatalogCache.Tables(accountId, ...) / CatalogCache.Databases(accountId, ...) filters by account
**Proposed optimization:**
Add tenant-level filtering to the catalog logtail subscription path so each CN only receives and caches catalog entries for the tenants it actually serves.
### Key changes needed:
1. **Subscription-level filtering**: Extend logtail subscription protocol to carry tenant/account info, so TN can filter catalog entries before sending
2. **TN-side filtering**: When collecting logtail for mo_database/mo_tables/mo_columns, filter rows by account_id based on subscribing CN tenant scope
3. **CN-side awareness**: CN should know which tenants it serves and communicate this during subscription
4. **Dynamic tenant changes**: Handle cases where tenant-to-CN mapping changes
## Why is this needed?
In multi-tenant deployments with many tenants (hundreds or thousands):
1. **Memory waste**: Every CN caches catalog metadata for ALL tenants in CatalogCache, even if it only serves a subset. Each TableItem is a large struct with TableDef, Constraint, CreateSql, etc.
2. **Network bandwidth waste**: TN pushes all tenants catalog logtail to every CN, multiplying network traffic by CN count
3. **Replay overhead**: CN replays catalog entries for tenants it never serves, wasting CPU during startup and incremental updates
4. **GC pressure**: Larger CatalogCache means more work during GC cycles
5. **Scalability bottleneck**: Per-CN overhead grows linearly with tenant count regardless of actual tenant assignment
## Additional information
### Considerations:
- System account (account_id=0) catalog entries are needed by all CNs
- Should consider lazy loading as alternative or complementary approach
Contributor guide
Assessment
This issue has not been assessed yet.