matrixorigin / matrixorigin/matrixone
[Feature Request]: High-speed DUMP TABLE and LOAD TABLE
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same feature request?
- [X] I have checked the existing issues.
### Is your feature request related to a problem?
Large benchmark and production-like datasets are expensive to move through row-oriented export and re-ingestion. Internal validation workflows need a fast way to dump and restore complete tables, including TPCH and TPCC datasets, without rebuilding every row through SQL.
### Describe the feature you would like
Add experimental SQL statements for high-speed table transfer:
- `DUMP TABLE TO `
- `LOAD TABLE FROM `
- Optional metadata-only dumps.
The feature should copy or reuse immutable table objects whenever possible and preserve the state required for a correct restore:
- Main-table data, tombstones, and auxiliary index relations.
- Schema and index-topology validation.
- AUTO_INCREMENT state, including allocator-owned hidden columns.
- Transactional object installation and cleanup on failure.
- Local files and stage-backed object storage.
The restored table must contain the same committed rows as the source, and subsequent inserts and index operations must continue correctly.
### Describe implementation you have considered
Use a bounded, versioned manifest to describe relation topology, object statistics, checksums, and relation-level metadata. Prefer provider-side object copies and install existing object metadata directly instead of decoding and re-encoding rows.
LOAD TABLE should validate the complete manifest before mutation, serialize object installation with table locks, submit data and tombstone objects in one transaction, and restore allocator state after object submission.
### Documentation, Adoption, Use Case, Migration Strategy
Initial scope is an experimental internal feature used by CI and engineering workflows.
Primary validation scenarios:
- TPCH table dump/load for analytical datasets.
- TPCC full-dataset dump/load, including AUTO_INCREMENT tables.
- Fast preparation of repeatable test datasets.
- Metadata-only copies for schema and topology validation.
Adoption can begin behind the existing experimental SQL surface. Compatibility guarantees and support for untrusted artifacts can be designed separately if the feature becomes public.
### Additional information
The initial implementation and TPCC validation are tracked by the associated pull request. A TKE experiment successfully round-tripped all ten TPCC tables and verified that inserts after LOAD TABLE continued above the restored AUTO_INCREMENT high watermark.
Contributor guide
Assessment
This issue has not been assessed yet.