A tagged version permanently prevents `cleanup_old_versions` from reclaiming newer data files

Description

When listing unreferenced files to delete, files are restricted to only those modified before the earliest_retained_manifest_time. Because a tagged version is always retained, a tag on a version that predates the cleanup retention window freezes the cleanup scan window at that tagged version's timestamp indefinitely. Any data file written after it is never a deletion candidate. However, versions referencing these files are deleted (their manifests are removed), such that the data files become unreadable and unreclaimable.

This contradicts the documented behaviour:

//! * Unreferenced data files - If a data file is not referenced by any
//!   fragment in a valid manifest file then it will be deleted.
...
//! If the file is referenced by at least one manifest (even if that manifest
//! is old and being deleted) then we assume it is not part of an ongoing
//! operation and can be safely deleted.
//!
//! If the data is not referenced by any manifest then we look at the age of
//! the file.  If the file is at least 7 days old then we assume it is probably
//! not part of any ongoing operation and we will delete it.

It is also a regression: before #4614 the listing was filtered by the caller's before cutoff (i.e. now - older_than), which advances on every run.

Output of the reproducer:

tag on a version outside of the cleanup retention window: False
  versions [1, 2, 3, 4, 5, 6, 7, 8] -> [8]
  stats.old_versions       7
  stats.data_files_removed 6
  unreferenced files left  0

tag on a version outside of the cleanup retention window: True
  versions [1, 2, 3, 4, 5, 6, 7, 8] -> [2, 8]
  stats.old_versions       6
  stats.data_files_removed 0
  unreferenced files left  4

Steps to reproduce

import datetime
import os
import tempfile
import lance
import pyarrow as pa

from pathlib import Path


def unreferenced(path: str) -> set[str]:
    ds = lance.dataset(path)
    live = {
        df.path
        for row in ds.versions()
        for frag in lance.dataset(path, version=row["version"]).get_fragments()
        for df in frag.metadata.files
    }
    return set(os.listdir(f"{path}/data")) - live


for tag_old_version in (False, True):
    with tempfile.TemporaryDirectory() as tmp:
        table = pa.table({"i": pa.array(range(64))})
        path = (
            Path(tmp) / "tagged.lance"
            if tag_old_version
            else Path(tmp) / "untagged.lance"
        )

        ds = lance.write_dataset(table, path)  # v1
        for _ in range(5):
            ds = lance.write_dataset(table, path, mode="append")  # v2-8

        if tag_old_version:
            ds.tags.create("pin", 2)  # the ONLY difference between the two cases

        ds.optimize.compact_files()  # pre-compaction data files become unreferenced

        versions_before = [row["version"] for row in lance.dataset(path).versions()]
        stats = lance.dataset(path).cleanup_old_versions(
            older_than=datetime.timedelta(0),
            delete_unverified=False,
            error_if_tagged_old_versions=False,
        )
        versions_after = [row["version"] for row in lance.dataset(path).versions()]

        print(
            f"tag on a version outside of the cleanup retention window: {tag_old_version}"
        )
        print(f"  versions {versions_before} -> {versions_after}")
        print(f"  stats.old_versions       {stats.old_versions}")
        print(f"  stats.data_files_removed {stats.data_files_removed}")
        print(f"  unreferenced files left  {len(unreferenced(path))}\n")

Expected behavior

Even when a tag is present on a dataset that predates the cleanup retention window, data files associated with versions being cleaned up should be removed, and no unreferenced files should be left:

tag on a version outside of the cleanup retention window: True
  versions [1, 2, 3, 4, 5, 6, 7, 8] -> [2, 8]
  stats.old_versions       6
  stats.data_files_removed 4
  unreferenced files left  0

Lance version

10.0.0

Language binding

Python

Environment

macOS 26.6.1, arm64, local

Logs / traceback

Start in rust/lance/src/dataset/cleanup.rs around the unreferenced-file listing at line 691, then read the cleanup behavior documented near line 13. Run the Python reproducer from the issue with and without the old tag. Done means cleanup removes the expected unreferenced data files while retaining the tagged version and leaves none behind.