lance-format / lance-format/lance
Calling list_indices results in "v2 format has no concept of row groups"
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
From Discord (kohlerbear):
Calling list_indices results in the following error:
>>> a=d.list_indices()
thread '<unnamed>' panicked at /home/runner/work/lance/lance/rust/lance-index/src/scalar/lance_format.rs:154:9:
not implemented: v2 format has no concept of row groups
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/root/.local/lib/python3.10/site-packages/lance/dataset.py", line 221, in list_indices
return self._ds.load_indices()
pyo3_runtime.PanicException: not implemented: v2 format has no concept of row groups
Any guidance on how to further debug this? This is on lancedb 0.24.3.
What I'm ultimately interested in: I'd like to understand my scalar index stats. I had a script I was using to enumerate index names, but I can also just use a direct call to dataset.stats.index_stats(). That results in a different error:
>>> print(d.stats.index_stats("occlusion_idx")["num_unindexed_rows"])
...
thread 'lance_background_thread' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-data-52.2.0/src/transform/utils.rs:42:56:
offset overflow
thread '<unnamed>' panicked at /home/runner/work/lance/lance/rust/lance-encoding/src/decoder.rs:1361:65:
called Result::unwrap() on an Err value: JoinError::Panic(Id(578), "offset overflow", ...)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/root/.local/lib/python3.10/site-packages/lance/dataset.py", line 2922, in index_stats
index_stats = json.loads(self._ds.index_statistics(index_name))
pyo3_runtime.PanicException: called Result::unwrap() on an Err` value: JoinError::Panic(Id(578), "offset overflow", ...)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with rust/lance-index/src/scalar/lance_format.rs:154 and the Python entry points Dataset.list_indices and stats.index_stats. Reproduce both reported failures on lancedb 0.24.3, then trace the v2 format and index-statistics paths; done means listing indices and retrieving scalar index statistics no longer panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100