cleanup_old_versions does not protect data files referenced via base_paths from another dataset (shallow-clone vacuum-on-source hazard)

Summary

cleanup_old_versions can delete data files that another dataset still references through base_paths. The cross-branch protection that exists today is within-dataset and descendant-only, so a shallow clone at a separate dataset root is not protected: running cleanup on the source of a shallow clone can orphan the clone's references, and the next read of those rows fails with a not-found error. This is the same hazard Delta Lake documents for running VACUUM on a shallow-clone source.

What's protected today

When cleaning a branch, retain_branch_lineage_files retains files that descendant branches reference: find_referenced_branches → collect_referenced_versions returns the branches whose lineage extends the current one (children), and for each it keeps any fragment whose base_id resolves to a BasePath inside the same dataset — the guard is base_path.path == self.dataset.uri (rust/lance/src/dataset/cleanup.rs). clean_referenced_branches defaults to false. So the parent→child, same-dataset case is covered: a parent branch's files still needed by a child branch are retained.

Not protected

1. Cross-dataset shallow clones (live today). shallow_clone writes a separate dataset at an arbitrary URI whose manifest references the source via base_paths — Manifest::shallow_clone inserts the source as a BasePath and stamps each fragment's base_id (rust/lance-table/src/format/manifest.rs). The source's cleanup_old_versions only enumerates self.dataset.branches() and only retains bases where base_path.path == self.dataset.uri, so it has no knowledge of an external clone. Cleaning the source deletes files the clone resolves through data_file_dir_for_base (rust/lance/src/dataset.rs), and the clone's next read of those rows fails. (The tags_and_branches guide states cleanup "does not delete files still referenced by any branch" — that holds for in-dataset branches, not for a cross-dataset clone.)
2. Ancestor-direction references (forthcoming with lance-format/lance#7263**).** Branch merge / rebase (lance-format/lance#7263) grafts a source branch's fragments into the target's base_paths. When the target is an ancestor — fast-forwarding/promoting into main, or rebasing onto a parent — the grafted files physically live in the source (descendant) tree but are referenced by the ancestor. The current walk is parent→child only, so cleaning or deleting the source branch would orphan the ancestor's references. This direction isn't reachable today (merge doesn't exist yet); flagging it so lance-format/lance#7263 can land GC-safe.

Repro (case 1, today)

1. shallow_clone dataset B from dataset A (separate URIs); B's manifest references A's data files via base_paths.
2. Run cleanup_old_versions on A with those versions no longer referenced by A's own lineage.
3. Read from B → the rows backed by A's now-deleted files fail with a not-found error.

Proposed direction

Make file retention reachability-complete over base_paths: a data file is live if any retained manifest — in any branch (both lineage directions) and any registered cross-dataset base — references it through base_id, not only descendants within the same dataset URI. Concretely, extend the referenced-file walk to follow base_paths regardless of the base_path.path == self.dataset.uri guard and regardless of lineage direction; or, short of that, document the constraint and gate the unsafe deletion behind an explicit opt-in. This is a prerequisite for lance-format/lance#7263 to be GC-safe and complements the branch-delete side in lance-format/lance#7185.

Related: lance-format/lance#7185, lance-format/lance#7263, #4858, lance-format/lance#7133.

Start in rust/lance/src/dataset/cleanup.rs, then trace Manifest::shallow_clone in rust/lance-table/src/format/manifest.rs and data_file_dir_for_base in rust/lance/src/dataset.rs. Map how cleanup discovers referenced files and how base_paths resolve across dataset roots and lineage directions. Done means cleanup cannot remove files still referenced by retained manifests, with coverage for the shallow-clone scenario.