listing FTS indexes is memory-intensive

Here is a quick example:

#!/usr/bin/env python3
"""Reproducer: list_indices() OOMs with large FTS index."""

import lance
import pyarrow as pa
import random
import string
import shutil
import threading
import time
import psutil
import os
from contextlib import contextmanager
import gc

PATH = "/tmp/fts_oom_test.lance"
NUM_ROWS = 200_000
WORDS_PER_ROW = 20


@contextmanager
def monitor_memory(label):
    gc.collect()
    process = psutil.Process(os.getpid())
    max_rss = process.memory_info().rss
    stop = threading.Event()

    def poll():
        nonlocal max_rss
        while not stop.is_set():
            max_rss = max(max_rss, process.memory_info().rss)
            time.sleep(0.01)

    thread = threading.Thread(target=poll)
    thread.start()
    start = time.time()
    try:
        yield
    finally:
        stop.set()
        thread.join()
        elapsed = time.time() - start
        print(f"{label}: {elapsed:.1f}s, peak RSS {max_rss / 1e9:.2f} GB")


shutil.rmtree(PATH, ignore_errors=True)

with monitor_memory("Creating dataset"):
    texts = [' '.join(''.join(random.choices(string.ascii_lowercase, k=12)) for _ in range(WORDS_PER_ROW)) for _ in range(NUM_ROWS)]
    table = pa.table({"text": texts})
    ds = lance.write_dataset(table, PATH)

with monitor_memory("Creating FTS index"):
    ds.create_scalar_index("text", index_type="INVERTED")

with monitor_memory("Calling list_indices()"):
    ds.list_indices()

Output:

Creating dataset: 3.3s, peak RSS 0.38 GB
Creating FTS index: 43.8s, peak RSS 4.55 GB
Calling list_indices(): 0.0s, peak RSS 3.82 GB

It is unexpected that listing the index takes a large fraction of the memory required to build it. It is also very slow on remote storage.

Start with the provided Python reproducer and the list_indices() entry point, then compare its memory behavior with FTS index creation on local and remote storage. Done means listing the index no longer consumes a large fraction of index-build memory and is no longer unexpectedly slow on remote storage.