Row id mapping follow-up tasks

This is a laundry list of things that came up during the "remap-row-ids-on-compaction" implementation that I felt were useful but not essential and so I left them for follow up.

Using reserved fragments is potentially dangerous

During a commit operation, if the user provides (in the transaction object) a fragment id that is non-zero then we just blindly assume that they have reserved this id. We should make a pass through the existing manifest and make sure that the id is not already in use.

Shrink index on remap instead of using tombstones

Currently, we keep the index the exact same size as it was before, by replacing the id of deleted rows with a tombstone. This means that we have to filter out these tombstones during a later PQ search which could have some performance penalty. Instead, we could go ahead and shrink the indices as we are remapping them.

Better algorithm for detection of missing row ids

In src/dataset/optimize.rs there is an iterator MissingIds which wraps another iterator of row ids and generates any ids that are missing from the provided range. Instead of making this iterator a proxy of a row-id iterator we could have the iterator scan the extracted row ids treemap directly. This would allow us to potentially, and quickly, skip entire ranges of row ids. See https://github.com/lancedb/lance/pull/1350#discussion_r1344895817

Find a way to still support multiple tasks as part of reindexing

Currently, if there are multiple tasks that would all affect the same index, then we only allow one of those tasks to run. This is a bit odd as it means you might have to run compaction multiple times to finally get everything compacted. One way we could work around this is to do the remapping at commit time, instead of when we are running tasks. This would mean we need to JSON serialize the remapped row vectors but that might not be a real problem. Another way to work around this is to add support for multi-file indices (which we might need for incremental updates anyways) and then the remap tasks could each take the rows they modify and split them out to a new partial index (instead of remapping the existing one). These could then be compacted together or left partial.

Start with the remap-row-ids-on-compaction implementation and src/dataset/optimize.rs, especially the MissingIds iterator and reindexing flow. The issue lists four separate follow-ups, so first determine which one is in scope and inspect the linked pull request discussion for the missing-ID algorithm. Done requires an agreed design and implementation for the selected follow-up, with behavior verified against the affected compaction or indexing paths.