Stability: index stability of Lance under ultra-large table scenarios

We maintain an extremely large Lance table, with roughly 100 million new rows added daily. When the table size reached around 4–5 billion rows, a series of index-related issues began to surface. This issue serves as an umbrella to document the indexing problems we encountered with large-scale tables and the fixes we implemented, in the hope that it can help the community and others exploring ultra-large datasets.

BloomFilterIndex

• Bug: bloomfilter index overflow when too many rows. https://github.com/lance-format/lance/pull/6522
• Performance: perf: parallel create/update bloomfilter. https://github.com/lance-format/lance/pull/6561

BitmapIndex

• Performance: support distributed bitmap index build. https://github.com/lance-format/lance/issues/6599
• Feat: enable large bitmap index for large table. https://github.com/lance-format/lance/pull/6642

BTreeIndex

• Performance: make btree and bitmap null tracking optional in scalar search. https://github.com/lance-format/lance/pull/6614

FTS

• OOM: introduce id-only load partition to avoid oom in fts. https://github.com/lance-format/lance/pull/6751