lance-format / lance-format/lance
feat: add file_writer_options for lance.write_dataset and LanceFragment.create
Open
Nobody has claimed this yet.
feature
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The file_writer_options can control the page size of lance file.
But now it can't be set in dataset and fragment API.
Add it into lance.write_dataset as:
dat = {
"foo": [1, 3],
"bar": ["one", "three"],
}
file_writer_options = {"data_cache_bytes": "4096", "max_page_bytes": "4096"}
ds = lance.write_dataset(dat, tmp_path, file_writer_options=file_writer_options)
Add it into LanceFragment.create as:
df = pd.DataFrame({"a": [1, 2, 3, 4, 5]})
file_writer_options = {"data_cache_bytes": "4096", "max_page_bytes": "4096"}
frag = LanceFragment.create(tmp_path, df, file_writer_options=file_writer_options)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Python definitions of lance.write_dataset and LanceFragment.create, then trace how their arguments reach the dataset and fragment writers. Add support for file_writer_options in both entry points and verify that the supplied options are honored when creating datasets and fragments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100