lance-format / lance-format/lance

Replace `list_indices()` and `index_statistics()` with `get_indexes` API

Open
#4,804 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Desired behavior changes:

  1. get_indexes() will return one entry per index name (not individual deltas, though they should still be accessible).
  2. Should be able to get statistics about coverage (num_indexed_rows and num_unindexed_rows) without any additional IO. This should be separate from index-specific information that is contained in the index file itself. Many use cases call index_stats just to get this information, but pay a performance penalty because it needs to load the index file. (See: https://github.com/lancedb/lance/issues/4620)
  3. Make it possible to get index type without having to open the index.
  4. Make it possible to inspect the index configuration (for example, distance type for ANN indices)

Usage

let dataset = ...;

let indexes = dataset.get_indexes().await?;

let ann_index = indexes.iter().find(|idx| idx.name == "vector_idx").unwrap();

// Get name
ann_index.name

// Get index type (should be immediately available)
ann_index.index_type

// Get coverage statistics (calculated on demand, but no IO needed)
ann_index.coverage_stats().num_unindexed_rows

// Get created and updated at
ann_index.created_at()
ann_index.updated_at()

// Get configuration (Maybe no IO needed?)
ann_index.config()

// Get index-specific stats (needs to be loaded from index file)
ann_index.index_stats().await

API

impl Dataset {
    async fn get_indexes(&self) -> Vec<Index<'_>>;
}


struct Index<'a> {
    pub name: &'a str,
    pub index_type: &'a str, // TODO: what should this type be?
    pub columns: &[&str],
    metadatas: Vec<&'a IndexMetadata>,
    ds: &'a Dataset,
}

impl Index<'_> {
    fn deltas(&self) -> &[&IndexMetadata];

    fn coverage_stats(&self) -> IndexCoverageStats {}

    fn created_at(&self) -> Instant {
        todo!("Get the min created_at from all index metadatas")
    }

    fn updated_at(&self) -> Instant {
        todo!("Get the max updated_at from all index metadatas")
    }

    fn config(&self) -> Config {}

    /// Retrieve index-specific statistics
    async fn index_stats(&self) -> Result<Box<dyn IndexStats>> {}
}

struct CoverageStats {
    pub num_rows_indexed: usize,
    pub num_unindexed_rows: usize,
}

TODO

  • Figure out what we we want the index_type parameter to be, given existence of extensions.
  • Figure out how we store the index configuration

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing list_indices() and index_statistics() APIs and their index metadata handling. Use the proposed Dataset::get_indexes() and Index methods as the target surface, then resolve the open questions around index_type and configuration storage. Done means indexes are grouped by name while retaining deltas, coverage is available without extra I/O, and index-specific statistics remain separately loadable.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.