lance-format / lance-format/lance
Feature request: error handling for `push_uri` API for error handling
Open
Nobody has claimed this yet.
feature
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Hi team, I'm heavily doing dataset conversion in my daily work, for example, convert open dataset from huggingface to lance.
One pain point that I have is
- I prefer to use
push_uriandIngestionmode for blob download ingestion, which internally decides blob storage option based on size and handles streaming download - But more often than not, blob download fails because of all types of reasons; for example, HTTP response
content-lengthreports incorrect number of bytes, the URL link doesn't work any more, DNS resolution failure, etc - In the current implementation, one single missing blob will fail the whole write process, which usually takes several hours long
- To unblock myself, I have to do all the above work myself (download at application side and call
push_blob), but I'm wondering if there's any chance lance could provide better support for error handling? For example, provides the ability to get users customize their own error handling logic, say, log and continue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the push_uri API's Ingestion-mode blob download path and how failures currently abort a write. Clarify the contract for customizable handling, including whether logging and continuing should be supported for incorrect content-length, unavailable URLs, and DNS failures. Done means the supported error behavior and its scope are specified for users.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100