matrixorigin / matrixorigin/matrixone
[Feature Request]: datalink — snapshots version the reference, not the referenced file bytes
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
**Version:** `8.0.30-MatrixOne-v3.0.11` (MatrixOne v3.0.11)
A `SNAPSHOT` of a table with a `datalink` column versions the **reference string**, not the **content** of the referenced external file. After a snapshot, overwriting the external object makes time-travel reads return the **new** bytes, so the historical state is not reproducible at the byte level.
**Reproduce**
```sql
CREATE STAGE st URL='s3://bucket/prefix/' CREDENTIALS={...};
CREATE TABLE docs(id INT PRIMARY KEY, ref datalink);
INSERT INTO docs VALUES (1, CAST('stage://st/doc.txt' AS datalink)); -- file content = "v1"
CREATE SNAPSHOT s FOR TABLE db docs;
-- now overwrite s3://bucket/prefix/doc.txt with "v2" out of band
SELECT load_file(ref) FROM db.docs {snapshot='s'} WHERE id=1; -- returns "v2", not "v1"
```
**Request**
Please document this behavior clearly, and consider an option to content-version datalink-referenced files (e.g. pin an object/content hash at snapshot time) so historical snapshots are byte-reproducible.
---
_Found while evaluating MatrixOne git4data / SQL as a data-versioning & agent-trace backend. Version: `8.0.30-MatrixOne-v3.0.11` (MatrixOne v3.0.11)._
## Testing background
Found while building a **multimodal dataset catalog** (images / video / documents referenced from a table via `STAGE` + `datalink`) and a lakeFS+MatrixOne integration, testing whether a `SNAPSHOT` makes a dataset version **byte-reproducible**. It does not: the snapshot pins the datalink *reference string*, and overwriting the external object out of band makes time-travel reads return the new bytes.
## Why it matters (expected purpose)
For multimodal training we need **byte-reproducible dataset versions** (a past snapshot should resolve to the exact file content used then). Today only the reference is versioned, so external-file edits silently change historical reads. Please document this clearly and/or offer an option to content-version datalink targets (e.g. pin a content hash / object version at snapshot time).
Contributor guide
Assessment
This issue has not been assessed yet.