matrixorigin / matrixorigin/matrixone
[Feature Request]: Add a native history-skipping latest-per-series execution path
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Motivation
MatrixOne has a correct and reasonably fast `max_by(value, order, tie)` aggregate, but latest-per-series still performs a complete historical scan. With a stable 10,000-series cardinality, latency and scanned rows grow linearly from 1M to 10M points even when the table is physically declared:
```sql
CLUSTER BY(device_id, event_ts)
```
Professional time-series engines normally provide a latest/last-point execution path that exploits designated time order, series dictionaries/indexes, block metadata, or maintained latest-state summaries. MatrixOne currently has no storage/scan operator that can stop once the newest point for each requested series is known.
This issue requests a native, deterministic latest-per-series capability. It is distinct from optimizing the generic `ROW_NUMBER` operator: even the purpose-built `max_by` path currently scans all history.
## Environment
- Repository: `matrixorigin/matrixone`
- Branch: `main`
- Commit: `1c292cf20e113b296a3db2cb6a65e00af2ea2327`
- Host: Debian x86_64, `10.222.1.55`
- CPU allocation: 16 logical CPUs (`GOMAXPROCS=16`)
- SQL endpoint: isolated quickstart at `127.0.0.1:6001`
- Server version: `8.0.30-MatrixOne-v`
- Data state: flushed, warm repeated queries
## Harness
Three tables use the same schema, `CLUSTER BY(device_id,event_ts)`, 10,000 devices, and deterministic timestamps/ties. Only historical depth changes:
| Table | Rows | Series | Points/series |
|---|---:|---:|---:|
| `p1m` | 1,000,000 | 10,000 | 100 |
| `p5m` | 5,000,000 | 10,000 | 500 |
| `p10m` | 10,000,000 | 10,000 | 1,000 |
Example construction:
```sql
CREATE TABLE p10m (
device_id INT,
event_ts DATETIME,
seq BIGINT,
value BIGINT
) CLUSTER BY(device_id, event_ts);
INSERT INTO p10m
SELECT result % 10000,
DATE_ADD('2026-01-01', INTERVAL FLOOR(result / 10000) SECOND),
FLOOR(result / 10000),
result
FROM generate_series(0, 9999999, 1) g;
SELECT mo_ctl('dn', 'flush', 'ts_latest_scale.p10m');
```
Measured query:
```sql
SELECT COUNT(*), SUM(seq), SUM(value)
FROM (
SELECT device_id,
max_by(seq, event_ts, seq) AS seq,
max_by(value, event_ts, seq) AS value
FROM p10m
GROUP BY device_id
) q;
```
## Results
All samples verify exact count and checksums.
| Rows | Warm p50 | Warm max | `EXPLAIN ANALYZE` input rows | Input blocks |
|---:|---:|---:|---:|---:|
| 1M | 46.9ms | 50.2ms | 1,000,000 | 132 |
| 5M | 154.5ms | 156.8ms | 5,000,000 | 624 |
| 10M | 308.3ms | 329.0ms | 10,000,000 | 1,232 |
The series count and output size remain fixed, but the query reads every historical row. The 5M and 10M results are almost exactly linear full-scan scaling.
At larger retention depths (hundreds of millions/billions of points), this turns a current-state lookup into a historical analytical scan. Adding more history degrades dashboard latency even when no new series are added.
## Requested capability
Provide a native latest-per-series plan and execution path. The SQL surface could be either:
- an optimizer-recognized `max_by` / Top-1-per-group pattern; or
- an explicit time-series `LATEST ... PARTITION BY` interface.
The mechanism should exploit available physical properties rather than introducing an unconditional cache. Depending on table layout, valid approaches may include:
- reverse time-ordered block/object scanning with early stop after all requested series are found;
- cluster-key/block-zone-map aware last-block selection per series range;
- optional bounded maintained latest-state metadata with explicit ownership, recovery, update/delete, and memory/cardinality limits;
- combining persisted-object candidates with the unflushed tail, then applying deterministic `(event_ts,tie)` comparison once.
The implementation must not assume all series report recently. It needs an explicit fallback when rare/inactive series require older history.
## Semantic requirements
- Deterministic winner from `(order_key, tie_key)`, matching `max_by`.
- Correct with duplicate timestamps, NULL values, late/out-of-order writes, UPDATE, DELETE, and transaction snapshots.
- Correct across persisted objects plus unflushed tail/logtail.
- Correct after flush, compaction, restart, and hard-crash recovery.
- Filters on tenant/device/tag and time cutoff must be preserved.
- No unbounded per-series CN memory or process-global cache.
- Plans that cannot exploit physical ordering must fall back to the existing full aggregate.
## Acceptance criteria
- `EXPLAIN ANALYZE` shows a dedicated latest scan/operator or equivalent early-stop behavior.
- For dense active series, scanned rows are governed primarily by series count/recent density rather than total retained history.
- On the stated 1M/5M/10M harness, latency should remain near-flat or grow substantially sublinearly as points per series increase 10x.
- Exact results match full-scan `max_by` as an independent oracle for tail, flushed, late-write, delete/update, compaction, and restart states.
- Add benchmarks for balanced and skewed/rare-series distributions; the latter must verify the bounded fallback behavior.
- Observability reports candidate blocks/rows scanned, early-stop effectiveness, and fallback count without high-cardinality metric labels.
## Value
Latest device state, current quotes, most recent health/status, current counters, and last known sensor values are first-class time-series workloads. A native path would:
- decouple current-state latency from retention length;
- avoid forcing every deployment to maintain a separate latest-state table;
- make `CLUSTER BY` materially useful for latest queries;
- provide a fast implementation target for portable `ROW_NUMBER ... = 1` rewrites;
- close an important capability gap relative to specialized time-series databases.
## Existing-issue check
Repository searches for native latest-per-series, latest-per-group scan, `max_by` cluster optimization, and Top-1 storage execution found no matching issue. #26229 introduced `max_by` for deterministic aggregation but does not provide or track a history-skipping latest scan.
Contributor guide
Assessment
This issue has not been assessed yet.