Benchmark failure: merge-insert hash join exceeds bounded memory pool

The bench-python / benchmark job in lance-bench failed against Lance commit b0b631e1d8af0aa42addfea4466b9a9bb3b2be09 (Lance 10.1.0-beta.1).

Run: https://github.com/lancedb/lance-bench/actions/runs/30675599910
Job: https://github.com/lancedb/lance-bench/actions/runs/30675599910/job/91302067770

Six merge-insert benchmark cases fail because HashJoinInput cannot allocate within the configured 150 MB fair memory pool:

Resources exhausted: Additional allocation failed for HashJoinInput
HashJoinInput(can spill: false) consumed 137.3 MB, peak 137.3 MB.
Failed to allocate additional 15.3 MB ... 12.7 MB remain available for the total memory pool: fair(pool_size: 150.0 MB)
rust/lance/src/dataset/write/merge_insert.rs:1228:34

Affected cases:

• test_update_subset_row_fraction[100pct-v1_indexed]
• test_update_subset_projection[one_scalar-100pct-v1_indexed]
• test_update_subset_projection[ten_scalars-100pct-v1_indexed] (requests 174.0 MB)
• test_update_subset_projection[vector-100pct-v1_indexed] (requests 984.2 MB)
• test_update_subset_projection[vector_and_ten_scalars-100pct-v1_indexed] (requests 1150.6 MB)
• test_upsert_wide_full_schema[100pct-v1_indexed] (requests 1321.7 MB)

Reproduction: build the Python package at the SHA above, generate the CI benchmark datasets, then run:

pytest python/ci_benchmarks/benchmarks/test_merge_insert.py --benchmark-only

The v2 hash variants complete, while these v1 indexed 100% update/upsert variants fail. The allocation requests up to 1.3 GB and can spill: false suggest the merge-insert join no longer operates within the benchmark memory bound.

Start with rust/lance/src/dataset/write/merge_insert.rs:1228 and reproduce using python/ci_benchmarks/benchmarks/test_merge_insert.py with the benchmark-only command. Compare the failing v1 indexed cases with the completing v2 hash variants, focusing on HashJoinInput's bounded pool allocation and non-spilling behavior. Done means the affected merge-insert benchmarks complete within the configured 150 MB pool.