Epic: Stable Row IDs to production ready

This project tracks the work to bring Stable Row IDs—which have been experimental for some time—to be production ready. Not just something that we can optionally turn on, but something we are confident we can make the default.

There are several changes to make, discussed in depth in this doc: https://docs.google.com/document/d/1x6Xon1XNZ5XaXHekk5kU0Ujg0A9FCOSLAP1Ry9XTMx8/edit?tab=t.0

This supersedes previous projects:

• https://github.com/lance-format/lance/issues/2454
• https://github.com/lance-format/lance/issues/2307

Issues

• Bug fixes
  • https://github.com/lance-format/lance/issues/6734
  • https://github.com/lance-format/lance/issues/8466
  • https://github.com/lance-format/lance/issues/8851
  • https://github.com/lance-format/lance/issues/8848
• §5.1 Fragment Reuse Hardening
  • https://github.com/lance-format/lance/issues/8977
• §5.2 Secondary indexes store row addresses, not row IDs
  • https://github.com/lance-format/lance/issues/8085
• §5.4 Fast row-ID lookup via a BTree index on _rowid
  • https://github.com/lance-format/lance/issues/8850

Start by reading the linked design document and the outstanding child issues, especially 8977, 8085, and 8850, along with the listed superseded projects. Map the remaining bug fixes and sections 5.1, 5.2, and 5.4; the epic is done when Stable Row IDs are production-ready and suitable to enable by default.