Fragment.take fails on non-monotonic offsets: "Encoding I/O returned N results for M requested range chunks"

Fragment.take fails whenever the offsets are not in ascending order. The internal error asks for a bug report, so here it is.

Reproduced on pylance 11.0.0, macOS arm64 and Linux x86-64, against a dataset written by the same version. Deterministic, and independent of thread or worker count.

Reproduction

import tempfile
from pathlib import Path

import lance
import numpy as np
import pyarrow as pa

rows, width = 4_000, 1_024
rng = np.random.default_rng(0)
raw = rng.integers(0, 256, size=rows * width, dtype=np.uint8).tobytes()
table = pa.table({"blob": pa.array([raw[at * width:(at + 1) * width] for at in range(rows)], pa.binary())})

uri = str(Path(tempfile.mkdtemp()) / "repro.lance")
lance.write_dataset(table, uri, max_rows_per_file=rows, enable_stable_row_ids=True)
fragment = lance.dataset(uri).get_fragments()[0]

offsets = rng.permutation(rows)[:512]
fragment.take(np.sort(offsets).tolist(), columns=["blob"])                      # 512 rows, fine
fragment.take(np.sort(np.concatenate([offsets, offsets])).tolist(), ["blob"])   # 1024 rows, fine
fragment.take(offsets.tolist(), columns=["blob"])                               # raises

The last call raises:

OSError: Encountered internal error. Please file a bug report at
https://github.com/lance-format/lance/issues. Encoding I/O returned 1 results
for 512 requested range chunks, lance-file/src/io.rs:104:28

What we narrowed it to

So the requirement is monotonicity, not distinctness — a repeated offset is served fine, a single inversion is not.

It is not specific to variable-width or compressed columns. The same failure occurs on a plain int64 column, on binary, on large_binary, and with or without lance-encoding:compression: zstd.

It depends on the volume the take spans, which we assume decides whether the read is split into more than one range chunk. At 1,000 rows of 8-byte values, 512 shuffled offsets succeed; at 1,000 rows of 512-byte values the same shuffled offsets fail. Small takes always succeed — 2 and 4 shuffled offsets were fine in every shape we tried.

A second, more severe symptom appears when the column is highly compressible (we used identical all-zero values, presumably a different encoding path): instead of the OSError, a background thread panics and the task aborts.

thread 'lance_background_thread' panicked at
lance-encoding/src/encodings/logical/primitive.rs:728:46:
called `Option::unwrap()` on a `None` value

Why it matters

This is the read path for a random-access sampler. Any shuffled draw — which is what sampling without replacement produces — lands in the failing shape as soon as the batch is big enough, so it is not an edge case for a training data loader. The message also asks the caller to file a bug rather than telling them the offsets must ascend, which cost us a while to track down.

A caller can work around it by sorting the offsets and mapping the rows back afterwards, which is what we now do. If ascending order is in fact a precondition rather than a defect, it would help a lot to validate it and say so in the error, and to document it on Fragment.take — but the panic looks like a genuine bug either way.

Start with the Fragment.take read path and the reported locations in lance-file/src/io.rs:104:28 and lance-encoding/src/encodings/logical/primitive.rs:728:46. Reproduce the shuffled, descending, and duplicate-offset cases from the issue, including the large and highly compressible columns. Done means non-monotonic offsets no longer return the range-chunk error or panic, or are explicitly validated and documented if ascending order is required.