matrixorigin / matrixorigin/matrixone
[Feature] Tiered hot/cold/archive data retention policy (Snowflake DATA_RETENTION_TIME_IN_DAYS + Fail-safe style, extended with storage tiers)
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
Add a **tiered, temperature-aware data retention policy** to MatrixOne, where a table/database/account can declare how long data stays in each storage tier — **HOT → COLD → ARCHIVE** — and what is queryable / recoverable at each tier. The retention semantics should follow Snowflake's well-known `DATA_RETENTION_TIME_IN_DAYS` + Time Travel + Fail-safe model, extended with explicit hot/cold/archive storage tiers (which Snowflake keeps automatic/opaque) so users get SQL-level control over the cost/performance trade-off. This maps naturally onto MatrixOne's object-storage architecture (S3 / OSS / COS) and existing Snapshot / PITR / Stage features.
## Motivation
- Today MatrixOne retention is effectively binary: data is live, or you keep history via `PITR` / `SNAPSHOT`. There is no SQL way to say *"keep the last 7 days hot, the last 90 days on cheap cold object storage, and 2 years in deep archive"*.
- Large fact tables (logs, events, IoT, orders) have a clear access gradient: recent data is queried constantly (hot), older data occasionally (cold), and the long tail almost never but must be retained for compliance (archive).
- Cloud object storage already offers cheap tiers (S3 Standard-IA / Glacier / Deep Archive; Tencent COS Standard / INFREQUENT / ARCHIVE / DEEP ARCHIVE; Aliyun OSS IA / Archive / Cold Archive). MatrixOne is object-storage-native and could expose this as a first-class, declarative lifecycle.
## Prior art — Snowflake (exact syntax)
Snowflake controls historical retention with **Time Travel** + **Fail-safe**:
```sql
-- Time Travel retention window, settable at account / database / schema / table level
ALTER ACCOUNT SET DATA_RETENTION_TIME_IN_DAYS = 90;
CREATE DATABASE sales DATA_RETENTION_TIME_IN_DAYS = 30;
CREATE TABLE events (id INT, ts TIMESTAMP)
DATA_RETENTION_TIME_IN_DAYS = 7
MAX_DATA_EXTENSION_TIME_IN_DAYS = 14;
ALTER TABLE events SET DATA_RETENTION_TIME_IN_DAYS = 1;
-- Query / recover within the Time Travel window
SELECT * FROM events AT (OFFSET => -60*5); -- 5 minutes ago
SELECT * FROM events AT (TIMESTAMP => '2026-06-01 00:00:00'::timestamp);
SELECT * FROM events BEFORE (STATEMENT => '');
CREATE TABLE events_restored CLONE events AT (OFFSET => -3600);
UNDROP TABLE events;
```
- **Time Travel** (0–90 days, Enterprise) = self-service historical query/clone/undrop.
- **Fail-safe** = a fixed, *non-configurable* **7-day** period *after* Time Travel expires; only Snowflake support can recover from it. It is effectively an opaque "archive/last-resort" tier.
- Note: Snowflake does **not** expose hot/cold/archive storage classes via SQL — tiering is automatic and hidden. That is exactly the gap this request proposes MatrixOne improve on, while keeping Snowflake's familiar retention semantics.
## Proposed MatrixOne syntax
Declarative per-tier retention; each tier names a retention window and (optionally) the object-storage class / stage it lives on.
```sql
-- Table-level tiered retention
CREATE TABLE events (id BIGINT PRIMARY KEY, ts DATETIME, payload JSON)
RETENTION (
HOT '7d', -- local/primary fast storage; full perf; Time-Travel queryable
COLD '90d' STORAGE = 'cos_ia', -- moved to infrequent-access object storage; queryable (slower)
ARCHIVE '730d' STORAGE = 'cos_archive' -- deep archive; must RESTORE before query; compliance retention
);
-- Account / database defaults (inherited, Snowflake-style)
ALTER ACCOUNT SET RETENTION (HOT '3d', COLD '30d', ARCHIVE '365d');
CREATE DATABASE logs RETENTION (HOT '1d', COLD '30d');
ALTER TABLE events SET RETENTION (HOT '14d', COLD '180d', ARCHIVE '1095d');
SHOW RETENTION FOR TABLE events; -- inspect effective policy + per-tier row counts/size
-- Bring archived partitions back online before querying
RESTORE TABLE events FROM ARCHIVE WHERE ts < '2024-01-01';
-- STORAGE handles reuse the existing STAGE / object-storage credential mechanism, e.g.
CREATE STAGE cos_archive URL='s3://bucket/archive/'
CREDENTIALS={...} PROPERTIES={'STORAGE_CLASS'='ARCHIVE'};
```
Semantics:
- **HOT** — current behavior: on primary storage, full read/write performance, covered by existing Time Travel / `{snapshot=...}` / PITR.
- **COLD** — automatically transitioned (by a background SQL Task–like lifecycle job) to a cheaper object-storage class; still directly queryable, with higher latency; transparent to SQL.
- **ARCHIVE** — deep/cold-archive class (Glacier / COS ARCHIVE); not directly queryable; requires an explicit `RESTORE TABLE ... FROM ARCHIVE` (analogous to Snowflake Fail-safe, but self-service and SQL-driven). Longest/compliance retention; data past the ARCHIVE window is permanently purged.
- Tiers are **cumulative thresholds by age** (or by a user-chosen partition/time column).
## Why this fits MatrixOne
- MatrixOne is already object-storage-native and supports **Stage** over S3/OSS/COS, **Snapshot**, **PITR**, and a native **SQL Task** scheduler — the building blocks for an automatic lifecycle engine are present.
- Reusing `STAGE` + storage-class properties means COLD/ARCHIVE tiers can point at the same buckets users already configure.
- Gives MatrixOne a clear, differentiated story vs. Snowflake: **explicit, SQL-level hot/cold/archive control** instead of opaque automatic tiering, at potentially much lower storage cost.
## Open questions / discussion
1. Tier transition trigger: age-based on a designated time column vs. snapshot-age vs. last-access.
2. Whether COLD stays transparently queryable or also needs a lightweight "thaw".
3. Interaction with PITR/Snapshot windows (should HOT retention ≥ PITR range?).
4. Granularity: whole table vs. partition/block-level transition.
5. Reuse `DATA_RETENTION_TIME_IN_DAYS` keyword for the HOT tier to ease Snowflake migration.
## Environment / context
Raised while validating Git-for-Data / lifecycle capabilities on MatrixOne Cloud `8.0.30-MatrixOne-v3.0.11` and local main build (commit `0bfda60`). This is a **feature request**, not a bug.
Contributor guide
Assessment
This issue has not been assessed yet.