Row-granular concurrency for V3 deletion vectors on UPDATE/MERGE (refine validateNoConflictingDeleteFiles)

This proposes reducing spurious conflict aborts for concurrent UPDATE/MERGE on
V3 deletion-vector tables when the operations touched disjoint rows. It gauges
appetite before writing code, and is scoped to build on the delete-side work
already in flight (#17754). Happy to move discussion to dev@iceberg.apache.org
if preferred.

## Background

With one DV per data file, two concurrent row-level operations on the same data
file conflict at file/partition granularity even when the rows they delete are
disjoint. The DV design discussion already noted this tradeoff — a single DV per
file makes the conflict file-level rather than row-level:
- https://lists.apache.org/thread/plsqgnlb0f84y5l5wf09qqtfmzdxbsxv
- https://lists.apache.org/thread/l5f5tkogpzcyk9sptbbj7nxby9so9fjg

Two checks produce these aborts on V3 DV tables:
- **`validateAddedDVs`** (core; runs for any row delta that adds DVs, so
DELETE/UPDATE/MERGE alike) fails if a concurrent commit added a DV for the same
data file — purely at file granularity, regardless of whether the deleted
positions overlap.
- **`validateNoConflictingDeleteFiles`** (Spark; called for UPDATE and MERGE,
guarded only by command type — not by isolation level, so it fires under both
SNAPSHOT and SERIALIZABLE) prunes concurrently-added delete files by partition
and column-metric bounds, then fails if any survive. It never compares the
actual deleted rows, so two UPDATE/MERGEs on disjoint rows of the same
partition still conflict.

Prior work:
- **#17754** (open) merges concurrently-added DVs for pure-delete RowDelta
commits instead of failing — position deletes are commutative, so the union is
always safe. It applies only when the commit adds/removes no data files and
`validateNoConflictingDeleteFiles` is not enabled — i.e. the DELETE path.
- **#14613** raised row-level concurrency as a direction earlier but closed
without a design.
- Delta Lake documents row-level concurrency on deletion-vector tables, reducing
concurrent-write conflicts from file to row granularity
(https://docs.delta.io/latest/concurrency-control.html).

## What UPDATE/MERGE still need

#17754 unblocks DELETE. UPDATE and MERGE additionally trigger
`validateNoConflictingDeleteFiles`, which fails disjoint operations independently
of the DV-merge, so refining `validateAddedDVs` alone does not help them. Making
`validateNoConflictingDeleteFiles` row-granular is the missing, unclaimed piece
for UPDATE/MERGE, and is the focus of this issue.

## Proposal

**1. Make `validateNoConflictingDeleteFiles` row-granular.** Today it fails when
any concurrently-added delete file survives partition + bounds pruning. Refine it
so that, for a surviving DV on a data file in this operation's scope, it conflicts
only when the DV's newly-deleted positions actually overlap the rows this
operation touched. Two tiers, different cost and guarantee:
- **write-set (bitmap only):** compare against the rows this operation itself
modified (its own DV). Catches lost updates → snapshot isolation, reads no data.
- **read-set (value-exact):** compare against the rows this operation read (its
predicate evaluated against the file). Preserves serializable isolation, but
reads file contents at commit.

**2. Extend the DV-merge to UPDATE/MERGE.** #17754's union is unconditional
because pure deletes are commutative; an UPDATE is delete + insert, so the union
is only safe when the newly-deleted positions are disjoint. This reuses #17754's
merge machinery plus a position-overlap test — `(dv_concurrent ∩ dv_self) −
dv_base == ∅`; union if disjoint, fail if not — which is the position-granular
form of `validateAddedDVs`. DVs added by an overwrite (which mark rows that live
on in rewritten files) are excluded, as #17754 notes.

The resolution lives inside the `apply()`/`validate()` cycle, which already
re-runs against a refreshed base on each retry of the commit loop
(`onlyRetryOn(CommitFailedException)`): the disjoint case unions and proceeds
(no exception), the overlapping case fails, and a concurrent commit landing
meanwhile re-runs the resolution against the new base. No separate retryable path
around the validation is needed — the disjoint case simply stops throwing.

## Conditions for soundness

- **One DV per file preserved:** disjoint results union into a single DV, never a
second parallel DV for the same file.
- **Position stability:** positions are comparable only if the data file is
unchanged. A concurrent compaction / copy-on-write that rewrote or removed the
file makes positions meaningless → fail (`validateDataFilesExist` /
`validateDeletedFiles` already guard this).
- **DV-only:** overlap requires both deletes to be DVs on the same file. An
equality delete or legacy position-delete file falls back to a conservative
failure.
- **Added rows are not ignored:** an UPDATE is delete-old + insert-new; the
insert/phantom dimension stays with the added-files check (follow-on A), so
serializable isolation is unaffected by the DV work.

## Additive follow-ons (sequenced)

Same best-effort principle: reconcile only when the loser's outcome is provably
reconstructable from DV positions or reader statistics; otherwise fall back to
today's abort.

**A. Added-files value-exact.** `validateNoConflictingDataFiles` /
`validateAddedDataFiles` use inclusive min/max metrics, so a serializable
UPDATE/MERGE false-fails on predicates bounds cannot model (e.g. `col % 7 = 3`).
Evaluating the real predicate against the surviving added files removes these
false failures, at the cost of a commit-time scan.

**B. Compaction ↔ DML.** A compaction and a DML that adds a DV to a file the
compaction rewrites do not reconcile today: whichever commits second fails
("found new position delete for replaced data file" on the compaction side,
"Cannot commit, missing data files" on the DML side), and `RewriteFiles` is built
to abort rather than remap. The cheap direction is when compaction commits first:
on the DML's re-apply against the refreshed base, its DV positions can be remapped
onto the rewritten file — old position `p` maps to `output_offset + (rank of p
among the rows compaction kept)`, computable from the source file's DV at
compaction's read snapshot, no data scan. This requires the rewrite to record a
small breadcrumb (per replaced file: the output file and the starting offset of
its rows), which Iceberg does not record today — `RewriteFiles` carries only flat
replaced/added file sets. It also relies on bin-pack preserving per-source order
(holds in practice: `DistributionMode.NONE` + sequential read); SORT/ZORDER
reorder rows and fall back to abort. Additive metadata plus turning the DML's
abort into a re-apply-and-remap — a separate, larger change than the row-granular
check above.

**Backlog (out of scope):** reclustering / ZORDER (rows permuted across files, no
derivable offset) and MERGE `NOT MATCHED BY SOURCE` (whole-target read, no
predicate to skip on) would need row identity plus a scan inside the conflict
path.

## Scope / non-goals

- MoR (deletion-vector) tables only; copy-on-write is out of scope — a rewrite
produces no DV to union, mask, or remap, so there is nothing to reconcile.
- No change to failing on genuine (row-overlap) conflicts.

## Configuration

An opt-in refinement plugs into the existing isolation infrastructure —
`write.{delete,update,merge}.isolation-level` and `IsolationLevel {SERIALIZABLE,
SNAPSHOT}` — rather than adding a parallel switch.

## Questions

1. Is the file/partition-level false-failure for disjoint UPDATE/MERGE worth
refining to row granularity?
2. For `validateNoConflictingDeleteFiles`: is the cheap write-set variant
(snapshot isolation, bitmap only) worth offering on its own, is only the
value-exact read-set variant (serializable) worthwhile, or should both be
offered and selected by the isolation-level config above?
3. Any prior art or earlier design decision I've missed beyond #17754 and #14613?

Start by reading the existing apply()/validate() cycle and the validateNoConflictingDeleteFiles and validateAddedDVs entry points, then review the delete-side merge work in #17754. Done means the project has agreed on the isolation-level behavior and row-overlap handling for UPDATE/MERGE while preserving failures for genuine conflicts; implementation and test files are not named.

Written by the indexing model from the issue text.