bug: orphaned staging manifests in _versions/ are never garbage collected

Problem

cleanup_old_versions can never collect an orphaned staging manifest. They accumulate in _versions/ indefinitely and require a storage-level delete to remove.

A staging manifest is written as the final manifest path plus a UUID suffix — make_staging_manifest_path (rust/lance-table/src/io/commit.rs:765-768):

pub(crate) fn make_staging_manifest_path(base: &Path) -> Result<Path> {
    let id = uuid::Uuid::new_v4().to_string();
    Path::parse(format!("{base}-{id}")).map_err(|e| Error::io_source(Box::new(e)))
}

so the filename is e.g. 00000000000000000042.manifest-cee4fbbb-eb19-4ea3-8ca7-54f5ec33dedc. There is exactly one . in that name, so Path::extension() returns the whole trailing string manifest-cee4fbbb-… — not manifest.

CleanupTask::classify dispatches on path.extension() (rust/lance/src/dataset/cleanup.rs:942). The value above matches no arm — not Some("manifest"), not lance/blob/arrow/bin — and falls through to the fallback at cleanup.rs:1097:

_ => Ok(None),

Ok(None) means "not a candidate for removal". The only prefix rule for the versions directory covers _versions/.tmp (cleanup.rs:898), which is a legacy layout no current writer emits (the only reference left is a test at cleanup.rs:3766).

Net: an orphaned staging manifest is invisible to cleanup at any age, under any policy.

How they get orphaned

The normal paths do delete them, so this is a crash/interruption residue rather than a steady-state leak:

• finalize_manifest deletes after the copy succeeds (rust/lance-table/src/io/commit/external_manifest.rs:689-701)
• commit_after deletes on the refused and losing paths (external_manifest.rs:1093, :1139)
• RenameCommitHandler consumes the staging object via the rename itself (rust/lance-table/src/io/commit.rs:1563)

An orphan is left behind when:

1. the process dies between writing the staging object and the delete/rename, or
2. lose_or_retain deliberately retains it because the commit outcome could not be resolved (external_manifest.rs:1132-1147) and nothing later resolves it.

Explicitly not affected

Manifests that ExternalManifestCommitHandler::commit_after leaves at a staging-shaped path on the winning path (external_manifest.rs:1080-1089) are real, live manifests recorded in the external store. Cleanup enumerates those through list_manifest_locations (which reads DynamoDB, external_manifest.rs:907-931) and deletes old ones by path directly from inspection.old_manifests (cleanup.rs:751-763), bypassing classify entirely. Those are collected correctly.

Impact

• Unbounded small-object accumulation in _versions/ on any dataset that has ever had an interrupted commit, with no supported way to remove them.
• Every _versions/ listing pays for them. They are filtered out post-listing by detect_scheme, so they consume listing page budget without contributing a result — see #8691.

Suggested direction

Give classify a rule for the versions directory rather than relying on the extension match: a file under _versions/ whose name is <manifest-name>-<uuid> and which is not the path recorded for any version could be treated like the existing CleanupFileKind::TemporaryManifest, i.e. removable once !maybe_in_progress.

Two things to be careful of:

• the retention at lose_or_retain is deliberate, so the age/in-progress gate matters more here than for _versions/.tmp
• the check must not catch a live commit_after manifest, which shares the same filename shape; comparing against the paths recorded in the external store is the distinguishing test