lance-format / lance-format/lance
Make open dataset with namespace less clunky
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Looking at the API, it seems like opening a table with a namespace is clunky compared to with a direct URI:
# With URI:
ds = lance.dataset('s3://bucket/path/to/table')
# With namespace:
namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset(namespace=namespace, table_id=['database', 'schema', 'table'])
I wonder if we can make it re-use the URI parameter, with maybe something like:
namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset("ns://database/schema/table", namespace=namespace)
Or we could use the $ separator, which I think we do elsewhere:
namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset("ns://database$schema$table", namespace=namespace)
Or honestly, it would be fine to just pass the table_id as the URI when namespace is also passed:
namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
data = lance.dataset(['database', 'schema', 'table'], namespace=namespace)
If we wanted to get really fancy, we could use locals() to resolve namespace identifiers in the scheme, so you don't have to explicitly pass it:
ns1 = lance.namespace.connect("rest", {"uri": "http://localhost:4099"})
other_namespace = lance.namespace.connect("rest", {"uri": "http://localhost:4100"})
# Use `locals()` to get available namespaces
data = lance.dataset("ns1://database/schema/table")
data = lance.dataset("other_namespace://database/schema/table")
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue centers on the Python lance.dataset entry point and its integration with namespace.connect. Read the existing handling for URI, table_id, and namespace arguments first; the issue is done only when one unambiguous form for namespace-backed tables is selected and its behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100