lance-format / lance-format/lance
Fragment take api: check indices in ascending order or support unordered?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Someone reports FileReader#take_rows checks the indices arg:
for i in range(len(indices) - 1):
if indices[i] > indices[i + 1]:
raise ValueError(
f"Indices must be sorted in ascending order for \
file API, got {indices[i]} > {indices[i + 1]}"
)
I have verified the Fragment#take, it also has similar issue:
def test_get_fragments(tmp_path: Path):
table = pa.Table.from_pydict({"a": range(100), "b": range(100)})
base_dir = tmp_path / "test"
lance.write_dataset(table, base_dir)
dataset = lance.dataset(base_dir)
fragment = dataset.get_fragments()[0]
assert fragment.count_rows() == 100
assert fragment.physical_rows == 100
assert fragment.num_deletions == 0
assert fragment.metadata.id == 0
head = fragment.head(10)
# tm.assert_frame_equal(head.to_pandas(), table.to_pandas()[0:10])
assert fragment.to_table() == table
taken = fragment.take([18, 20, 33, 53])
assert taken == pa.Table.from_pydict({"a": [18, 20, 33, 53], "b": [18, 20, 33, 53]})
# error case
# taken = fragment.take([20, 18, 53, 33])
# assert taken == pa.Table.from_pydict({"a": [20, 18, 53, 33], "b": [20, 18, 53, 33]})
The failed information is:
python/tests/test_dataset.py::test_get_fragments
thread 'lance_background_thread' panicked at /xxxx/lance/rust/lance-encoding/src/decoder.rs:1231:9:
assertion failed: indices.windows(2).all(|w| w[0] <= w[1])
Currently, it seems only Dataset#take does not require indices' orders.
The exception raised by Rust is ugly. It would be better to check indices in ascending order or support unordered?
Not sure if there is something blocking us from supporting an unordered index.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with FileReader#take_rows and Fragment#take, then compare their behavior with Dataset#take. Reproduce the case in python/tests/test_dataset.py and inspect the assertion in rust/lance-encoding/src/decoder.rs. Done means the chosen ordered-index behavior is consistent and produces a usable error or supports unordered indices, with tests covering it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend-api-design, data-engineering, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100