Blob API for URL

Background

Cloud providers offer model inference services tailored for multimodal data processing. Such services generally accept URLs as input to ingest data directly from external or cloud-based storage systems. To enable seamless integration with these inference platforms, we need to extend the Lance Blob API to support handling URLs as valid data references.

Blob V2 Design has already proposed supporting blob URI retrieval. This document aims to refine the URL handling component of this Blob API design.

Design

We will implement three core URL retrieval interfaces for the Lance Blob API:

• signed_url: Retrieves the presigned URL and range metadata of an object if the underlying storage supports it. This method is intended for scenarios where objects are not publicly accessible, as it generates a time-limited, temporarily accessible URL with embedded access permissions via presigning.
• public_url: Retrieves the public URL and range metadata of an object if both the storage supports it and the object has explicitly been configured with public-read permissions. This method is designed for objects that are intended to be accessed directly over the internet without any additional authentication.
• object_url: Generates an object URL based on the Lance Blob API's defined storage_prefix (a unique identifier for the target storage instance). This method is specifically tailored for Lance datasets stored in non-cloud, local storage systems like the file system (file protocol) or in-memory storage (memory protocol).

API (python)

class BlobFile(io.RawIOBase):
    def signed_url(
        self, expires_in_seconds: int = 3600
    ) -> Optional[tuple[str, tuple[int, int]]]:
        """Return a pre-signed URL and byte range for this blob.

        Parameters
        ----------
        expires_in_seconds:
            The expires duration in seconds of the URL.

        Returns
        -------
        url:
            The pre-signed URL for this blob.
        range:
            A ``(offset, length)`` pair in bytes describing where the blob
            payload is located inside the underlying object.

        Notes
        -----
        If the backend does not support pre-signed URLs for this blob, this
        method returns ``None``.
        """
        return self.inner.signed_url(expires_in_seconds)

    def public_url(self) -> Optional[tuple[str, tuple[int, int]]]:
        """Return a public URL and byte range for this blob.

        Returns
        -------
        url:
            The public URL for this blob.
        range:
            A ``(offset, length)`` pair in bytes describing where the blob
            payload is located inside the underlying object.

        Notes
        -----
        If the backend does not expose a public URL for this blob, this method
        returns ``None``.
        """
        return self.inner.public_url()

    def object_url(self) -> tuple[str, tuple[int, int]]:
        """Return an object URL and byte range for this blob.

        Returns
        -------
        url:
            The object URL for this blob.
        range:
            A ``(offset, length)`` pair in bytes describing where the blob
            payload is located inside the underlying object.

        Notes
        -----
        Object URLs are derived from the store prefix and path and are
        expected to be stable.
        """
        return self.inner.object_url()

PR

https://github.com/lance-format/lance/pull/5766

Start with the BlobFile entry point and its signed_url, public_url, and object_url interfaces, then read the linked Blob V2 Design document. Review PR #5766 before doing any work because the issue links to it. Done means the three URL retrieval interfaces cover the storage backends and return the documented URL and byte-range values.