lance-format / lance-format/lance
Replace `list_indices()` and `index_statistics()` with `get_indexes` API
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Desired behavior changes:
get_indexes()will return one entry per index name (not individual deltas, though they should still be accessible).- Should be able to get statistics about coverage (
num_indexed_rowsandnum_unindexed_rows) without any additional IO. This should be separate from index-specific information that is contained in the index file itself. Many use cases callindex_statsjust to get this information, but pay a performance penalty because it needs to load the index file. (See: https://github.com/lancedb/lance/issues/4620) - Make it possible to get index type without having to open the index.
- Make it possible to inspect the index configuration (for example, distance type for ANN indices)
Usage
let dataset = ...;
let indexes = dataset.get_indexes().await?;
let ann_index = indexes.iter().find(|idx| idx.name == "vector_idx").unwrap();
// Get name
ann_index.name
// Get index type (should be immediately available)
ann_index.index_type
// Get coverage statistics (calculated on demand, but no IO needed)
ann_index.coverage_stats().num_unindexed_rows
// Get created and updated at
ann_index.created_at()
ann_index.updated_at()
// Get configuration (Maybe no IO needed?)
ann_index.config()
// Get index-specific stats (needs to be loaded from index file)
ann_index.index_stats().await
API
impl Dataset {
async fn get_indexes(&self) -> Vec<Index<'_>>;
}
struct Index<'a> {
pub name: &'a str,
pub index_type: &'a str, // TODO: what should this type be?
pub columns: &[&str],
metadatas: Vec<&'a IndexMetadata>,
ds: &'a Dataset,
}
impl Index<'_> {
fn deltas(&self) -> &[&IndexMetadata];
fn coverage_stats(&self) -> IndexCoverageStats {}
fn created_at(&self) -> Instant {
todo!("Get the min created_at from all index metadatas")
}
fn updated_at(&self) -> Instant {
todo!("Get the max updated_at from all index metadatas")
}
fn config(&self) -> Config {}
/// Retrieve index-specific statistics
async fn index_stats(&self) -> Result<Box<dyn IndexStats>> {}
}
struct CoverageStats {
pub num_rows_indexed: usize,
pub num_unindexed_rows: usize,
}
TODO
- Figure out what we we want the
index_typeparameter to be, given existence of extensions. - Figure out how we store the index configuration
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing list_indices() and index_statistics() APIs and their index metadata handling. Use the proposed Dataset::get_indexes() and Index methods as the target surface, then resolve the open questions around index_type and configuration storage. Done means indexes are grouped by name while retaining deltas, coverage is available without extra I/O, and index-specific statistics remain separately loadable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100