lance-format / lance-format/lance

Calling list_indices results in "v2 format has no concept of row groups"

Open
#4,544 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-format enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

From Discord (kohlerbear):

Calling list_indices results in the following error:

>>> a=d.list_indices()
thread '<unnamed>' panicked at /home/runner/work/lance/lance/rust/lance-index/src/scalar/lance_format.rs:154:9:
not implemented: v2 format has no concept of row groups
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/root/.local/lib/python3.10/site-packages/lance/dataset.py", line 221, in list_indices
    return self._ds.load_indices()
pyo3_runtime.PanicException: not implemented: v2 format has no concept of row groups

Any guidance on how to further debug this? This is on lancedb 0.24.3.

What I'm ultimately interested in: I'd like to understand my scalar index stats. I had a script I was using to enumerate index names, but I can also just use a direct call to dataset.stats.index_stats(). That results in a different error:

>>> print(d.stats.index_stats("occlusion_idx")["num_unindexed_rows"])
...
thread 'lance_background_thread' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-data-52.2.0/src/transform/utils.rs:42:56:
offset overflow
thread '<unnamed>' panicked at /home/runner/work/lance/lance/rust/lance-encoding/src/decoder.rs:1361:65:
called Result::unwrap() on an Err value: JoinError::Panic(Id(578), "offset overflow", ...)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/root/.local/lib/python3.10/site-packages/lance/dataset.py", line 2922, in index_stats
    index_stats = json.loads(self._ds.index_statistics(index_name))
pyo3_runtime.PanicException: called Result::unwrap() on an Err` value: JoinError::Panic(Id(578), "offset overflow", ...)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with rust/lance-index/src/scalar/lance_format.rs:154 and the Python entry points Dataset.list_indices and stats.index_stats. Reproduce both reported failures on lancedb 0.24.3, then trace the v2 format and index-statistics paths; done means listing indices and retrieving scalar index statistics no longer panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.