matrixorigin / matrixorigin/matrixone

[Feature Request]: datalink — snapshots version the reference, not the referenced file bytes

Open
#24,555 0 comments 0 reactions 1 assignee Claimed by @ck89119 View on GitHub
deferred kind/feature
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

**Version:** `8.0.30-MatrixOne-v3.0.11` (MatrixOne v3.0.11)

A `SNAPSHOT` of a table with a `datalink` column versions the **reference string**, not the **content** of the referenced external file. After a snapshot, overwriting the external object makes time-travel reads return the **new** bytes, so the historical state is not reproducible at the byte level.

**Reproduce**
```sql
CREATE STAGE st URL='s3://bucket/prefix/' CREDENTIALS={...};
CREATE TABLE docs(id INT PRIMARY KEY, ref datalink);
INSERT INTO docs VALUES (1, CAST('stage://st/doc.txt' AS datalink)); -- file content = "v1"
CREATE SNAPSHOT s FOR TABLE db docs;
-- now overwrite s3://bucket/prefix/doc.txt with "v2" out of band
SELECT load_file(ref) FROM db.docs {snapshot='s'} WHERE id=1; -- returns "v2", not "v1"
```

**Request**
Please document this behavior clearly, and consider an option to content-version datalink-referenced files (e.g. pin an object/content hash at snapshot time) so historical snapshots are byte-reproducible.

---
_Found while evaluating MatrixOne git4data / SQL as a data-versioning & agent-trace backend. Version: `8.0.30-MatrixOne-v3.0.11` (MatrixOne v3.0.11)._

## Testing background
Found while building a **multimodal dataset catalog** (images / video / documents referenced from a table via `STAGE` + `datalink`) and a lakeFS+MatrixOne integration, testing whether a `SNAPSHOT` makes a dataset version **byte-reproducible**. It does not: the snapshot pins the datalink *reference string*, and overwriting the external object out of band makes time-travel reads return the new bytes.

## Why it matters (expected purpose)
For multimodal training we need **byte-reproducible dataset versions** (a past snapshot should resolve to the exact file content used then). Today only the reference is versioned, so external-file edits silently change historical reads. Please document this clearly and/or offer an option to content-version datalink targets (e.g. pin a content hash / object version at snapshot time).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.