lance-format / lance-format/lance

Feature request: error handling for `push_uri` API for error handling

Open
#9,362 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Hi team, I'm heavily doing dataset conversion in my daily work, for example, convert open dataset from huggingface to lance.
One pain point that I have is

  • I prefer to use push_uri and Ingestion mode for blob download ingestion, which internally decides blob storage option based on size and handles streaming download
  • But more often than not, blob download fails because of all types of reasons; for example, HTTP response content-length reports incorrect number of bytes, the URL link doesn't work any more, DNS resolution failure, etc
  • In the current implementation, one single missing blob will fail the whole write process, which usually takes several hours long
  • To unblock myself, I have to do all the above work myself (download at application side and call push_blob), but I'm wondering if there's any chance lance could provide better support for error handling? For example, provides the ability to get users customize their own error handling logic, say, log and continue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the push_uri API's Ingestion-mode blob download path and how failures currently abort a write. Clarify the contract for customizable handling, including whether logging and continuing should be supported for incorrect content-length, unavailable URLs, and DNS failures. Done means the supported error behavior and its scope are specified for users.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.