lance-format / lance-format/lance

Make open dataset with namespace less clunky

Open
#5,809 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-python
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Looking at the API, it seems like opening a table with a namespace is clunky compared to with a direct URI:

# With URI:
ds = lance.dataset('s3://bucket/path/to/table')

# With namespace:
namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset(namespace=namespace, table_id=['database', 'schema', 'table'])

I wonder if we can make it re-use the URI parameter, with maybe something like:

namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset("ns://database/schema/table", namespace=namespace)

Or we could use the $ separator, which I think we do elsewhere:

namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset("ns://database$schema$table", namespace=namespace)

Or honestly, it would be fine to just pass the table_id as the URI when namespace is also passed:

namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset(['database', 'schema', 'table'], namespace=namespace)

If we wanted to get really fancy, we could use locals() to resolve namespace identifiers in the scheme, so you don't have to explicitly pass it:

ns1 = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
other_namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4100"})

# Use `locals()` to get available namespaces
data = lance.dataset("ns1://database/schema/table")
data = lance.dataset("other_namespace://database/schema/table")

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue centers on the Python lance.dataset entry point and its integration with namespace.connect. Read the existing handling for URI, table_id, and namespace arguments first; the issue is done only when one unambiguous form for namespace-backed tables is selected and its behavior is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.