lance-format / lance-format/lance
Add "blobs" to the table format
@westonpace is already working on this.
Since Apr 26, 2024.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We are currently writing large values such as images and videos into the the data files themselves. Since the format is designed to handle large values this works reasonably well. However, for very large values (4/8 MB and above) this causes issues with compaction (both RAM required to run compaction and the total amount of I/O that must be performed).
We can work around this by using separate storage for blob files. The simplest approach is for this separate storage to be a nested Lance dataset. The nested dataset will be unversioned (all commits will be detached commits) and it will store blob fields.
Tasks:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.