matrixorigin / matrixorigin/matrixone

[Tech Request]: Tenant-Aware Catalog Logtail Subscription for CN Nodes

Open
#23,777 0 comments 0 reactions 1 assignee Claimed by @aptend View on GitHub
kind/tech-request
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Is there an existing issue for the same tech request?

- [x] I have checked the existing issues.

## Does this tech request not affect user experience?

- [x] This tech request doesn't affect user experience.

## What would you like to be added?

### Tenant-Aware Catalog Logtail Subscription for CN Nodes

Currently, when a CN starts up, it subscribes to the full logtail of system catalog tables (mo_database, mo_tables, mo_columns) via PushClient.subSysTables(). The TN sends all tenants catalog data to every CN, and the CatalogCache stores entries for all accounts in memory, only filtering by AccountId at query time.

**Current flow:**
1. CN calls subSysTables() -> subscribes to MO_DATABASE_ID, MO_TABLES_ID, MO_COLUMNS_ID (no tenant filter)
2. TN collects and pushes logtail for these tables containing all tenants data
3. CN applyToCatalogCache() inserts all rows into CatalogCache (btree of TableItem/DatabaseItem)
4. Query-time filtering: CatalogCache.Tables(accountId, ...) / CatalogCache.Databases(accountId, ...) filters by account

**Proposed optimization:**
Add tenant-level filtering to the catalog logtail subscription path so each CN only receives and caches catalog entries for the tenants it actually serves.

### Key changes needed:

1. **Subscription-level filtering**: Extend logtail subscription protocol to carry tenant/account info, so TN can filter catalog entries before sending
2. **TN-side filtering**: When collecting logtail for mo_database/mo_tables/mo_columns, filter rows by account_id based on subscribing CN tenant scope
3. **CN-side awareness**: CN should know which tenants it serves and communicate this during subscription
4. **Dynamic tenant changes**: Handle cases where tenant-to-CN mapping changes

## Why is this needed?

In multi-tenant deployments with many tenants (hundreds or thousands):

1. **Memory waste**: Every CN caches catalog metadata for ALL tenants in CatalogCache, even if it only serves a subset. Each TableItem is a large struct with TableDef, Constraint, CreateSql, etc.
2. **Network bandwidth waste**: TN pushes all tenants catalog logtail to every CN, multiplying network traffic by CN count
3. **Replay overhead**: CN replays catalog entries for tenants it never serves, wasting CPU during startup and incremental updates
4. **GC pressure**: Larger CatalogCache means more work during GC cycles
5. **Scalability bottleneck**: Per-CN overhead grows linearly with tenant count regardless of actual tenant assignment

## Additional information

### Considerations:
- System account (account_id=0) catalog entries are needed by all CNs
- Should consider lazy loading as alternative or complementary approach

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.