write_fragments(enable_stable_row_ids=True) silently produces fragments without row-id metadata, rejected on commit

Environment: pylance 9.0.0-beta.15.

write_fragments(..., enable_stable_row_ids=True) returns fragments whose row_id_meta is None — the flag is silently ignored. Committing those fragments onto a stable-row-id dataset (the only datasets the flag is for) is then rejected, far from the cause:

import tempfile
import lance
import pyarrow as pa
from lance.fragment import write_fragments

uri = tempfile.mkdtemp() + "/t.lance"
ds = lance.write_dataset(
    pa.table({"id": [1, 2, 3, 4], "v": [10, 20, 30, 40]}),
    uri, max_rows_per_file=2, enable_stable_row_ids=True)

new_frags = write_fragments(
    pa.table({"id": [5, 6], "v": [50, 60]}), uri, enable_stable_row_ids=True)
print([f.row_id_meta for f in new_frags])   # [None] -- flag silently ignored

all_frags = [f.metadata for f in ds.get_fragments()] + list(new_frags)
op = lance.LanceOperation.Merge(fragments=all_frags, schema=ds.schema)
lance.LanceDataset.commit(uri, op, read_version=ds.version)
# OSError: Invalid user input: All fragments must have row ids,
#   rust/lance-table/src/feature_flags.rs:57

Expected: either write_fragments honors the flag and emits row_id_meta so the fragments are committable, or it raises at call time if row-id assignment is out of scope for this API.

Impact (corrected after root-causing): the documented staged-write pattern write_fragments -> LanceOperation.Append -> commit works correctly on stable-row-id datasets -- Append assigns row-id sequences at commit time, which is the sound design (sequences must be allocated from the manifest's single next_row_id counter, which detached distributed writers cannot coordinate). The gap is Merge-specific, and the flag on write_fragments is misleading either way.

Root cause

enable_stable_row_ids has been a silent no-op on write_fragments since it was added (#3235): the flag lands in WriteParams, but its only consumers are validate_write (which overwrites it from the manifest on Append) and do_commit -- and write_fragments goes through execute_uncommitted_stream, which never reaches do_commit. The physical fragment writer never touches row_id_meta. Row ids are assigned at commit time by Transaction::assign_row_ids (transaction.rs:2968) in the Append/Update/Overwrite/Rewrite arms -- but not in the Merge arm (transaction.rs:2113), which also skips fragment-id assignment. The feature_flags.rs:57 backstop then rejects the mixed row-id state, far from the cause.

Two related findings:

• Committing the same write_fragments output via LanceOperation.Append succeeds and assigns row ids correctly (flag on or off -- it changes nothing).
• On a non-stable dataset, Merge with new fragments "succeeds" and silently commits duplicate fragment ids ([0, 0, 1]), since the Merge arm skips fragments_with_ids too. The resulting manifest is corruption-grade: fragment id is the upper half of every row address, so rows in the two id-0 fragments collide (to_table(with_row_id=True) returns duplicate _rowids; take on a colliding address returns one of two rows arbitrarily), and a subsequent delete("id = 1") -- one row -- left [1, 2, 3, 4, 5, 6] as [3, 4, 6, 6]: three rows silently destroyed and one duplicated by a single-row delete. Merge's documented contract says its fragments "must align with old ones at rows" (i.e. it was not designed to introduce fragments), but #6640 partially blessed new-fragment ids in Merge without adding either assignment step.

Fix shape: (1) deprecate or reject enable_stable_row_ids on write_fragments -- it can never be honored there by design; document that row ids are assigned at commit. (2) In the Merge arm, either reject fragments not present in the previous manifest ("Merge cannot introduce new fragments; use Append") -- which also fixes the silent duplicate-id corruption -- or mirror Append by running new fragments through fragments_with_ids + assign_row_ids (the existing Ordering::Equal branch already skips fragments with complete sequences). Verified still present on 9.0.0-beta.19 and current main.

Start with execute_uncommitted_stream for write_fragments, then compare the Merge arm in transaction.rs with Append, including fragments_with_ids and assign_row_ids at transaction.rs:2968. Inspect feature_flags.rs:57 and reproduce the stable and non-stable cases; done means Merge no longer creates invalid or duplicate fragment metadata and write_fragments makes its row-ID behavior explicit, with regression coverage.