lance-format / lance-format/lance
feat: add column ordering for pylance scanner
Open
Nobody has claimed this yet.
A-python
feature
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Add ordering for pylance scanner API
You can define four types of ColumnOrdering:
ColumnOrdering.asc_nulls_last("column_name")
ColumnOrdering.asc_nulls_first("column_name")
ColumnOrdering.desc_nulls_last("column_name")
ColumnOrdering.desc_nulls_first("column_name")
And you can use the ColumnOrdering in this API:
data = pa.table({"int_col": [2, 1, 3, None, 4], "str_col": ["a", None, "d", "b", "c"]})
dataset = lance.write_dataset(data, base_dir)
## asc null_last
ordering = ColumnOrdering.asc_nulls_last("int_col")
dataset.to_table(orderings=[ordering])
pa.Table.from_batches(dataset.to_batches(orderings=[ordering]))
dataset.scanner(orderings=[ordering]).to_table()
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python scanner API entry points named in the issue: ColumnOrdering, dataset.to_table, dataset.to_batches, and dataset.scanner. Verify how the four null-ordering variants should be represented and exercised with the provided integer and string examples; done means each ordering works consistently across all three APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100