matrixorigin / matrixorigin/matrixone

[Tech Request]: GC/reclamation for pinned datalink content-addressed store (CAS)

Open
#24,743 0 comments 0 reactions 1 assignee Claimed by @ck89119 View on GitHub
deferred kind/tech-request
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same tech request?

- [x] I have checked the existing issues.

### Does this tech request not affect user experience?

- [x] This tech request doesn't affect user experience.

### What would you like to be added ?

Add garbage collection / reclamation for the content-addressed store (CAS) that backs pinned datalinks.

Follow-up to #24555 / PR #24641. That PR introduced `datalink_pin()`, which freezes the bytes a datalink references into an immutable, per-account CAS keyed by content hash:

```
datalink_cas//

/
```

The store is intentionally **write-once and immutable** for byte-reproducibility, and that PR deliberately scoped *out* reclamation. As a result, a CAS blob is currently **retained indefinitely** — nothing reclaims it after the datalink values that referenced it are gone (rows deleted, table dropped, snapshot expired, or the whole account dropped). Over time this leaks storage.

This issue tracks designing and implementing reclamation. Two candidate mechanisms (not mutually exclusive):

1. **Account-scoped cleanup on `DROP ACCOUNT`** — drop the entire `datalink_cas//` prefix so a removed tenant does not leak its pinned blobs. Coarse-grained, simple, and safe (the namespace is already per-account).
2. **Reference-aware sweep** — mark blobs still reachable from live `datalink` columns and live snapshots, then delete the unreferenced rest. Required for the general case because a single content hash can be shared by many rows / snapshots (per-account dedup), so a blob must not be collected while *any* reference survives.

### Why is this needed ?

Without reclamation, pinned content accumulates in the SHARED file service forever and cannot be reclaimed even when no datalink references it, which is unsustainable for long-lived deployments and for tenants that pin large volumes of content.

### Additional information

- Feature / parent: #24555 (datalink content-versioning via content-hash pinning)
- Introducing PR: #24641
- CAS implementation: `pkg/datalink/cas.go` (`CASKey` / `CASPut` / `CASExists`; note there is currently **no** `CASDelete`/GC entry point)
- Design questions to resolve: reference discovery across tables + snapshots, concurrency vs. concurrent `datalink_pin()`, and whether reclamation runs as an online sweep, on DDL events (`DROP ACCOUNT` / `DROP TABLE`), or both.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.