lance-format / lance-format/lance-namespace-impls

Catalog impls access request fields as attributes but pylance's Rust bridge passes a DictWithModelDump (AttributeError on describe_table write path)

Open
#181 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
11
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Summary

When pylance writes a Lance table to a namespace-backed catalog (e.g. Glue) and its Rust storage-options provider calls back into the Python namespace impl's describe_table, it passes the request as a DictWithModelDump (a plain dict subclass exposing only model_dump(), no attributes). But GlueNamespace.describe_table (and the other catalog impls) access the request via attribute (request.load_detailed_metadata), which raises:

AttributeError: 'DictWithModelDump' object has no attribute 'load_detailed_metadata'

This breaks the write path (lance.dataset(...).write / lance_ray.write_lance) for every catalog impl in this package when driven through pylance's Rust namespace bridge.

Environment (a concrete, reproducing set)

  • pylance == 9.0.0 (pins lance-namespace>=0.8.5,<0.9)
  • lance-namespace == 0.8.6
  • lance-namespace-urllib3-client == 0.8.6
  • lance-namespace-impls == 0.4.1
  • lance-ray == 0.5.0

Root cause

pylance's PyO3 bridge (python/src/namespace.rs, present across 6.x–10.x) builds a request, serializes it to JSON, wraps it in a DictWithModelDump, and invokes the Python namespace impl's method with that dict. pylance's own wrappers (DirectoryNamespace/RestNamespace in lance/namespace.py) tolerate this because they call request.model_dump(). But the impls in this package access request fields as attributes.

describe_table in lance_namespace_impls/glue.py (v0.4.1, lines 353–358):

def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse:
    """Describe a table."""
    if request.load_detailed_metadata:          # <-- AttributeError on DictWithModelDump
        raise RuntimeError(
            "load_detailed_metadata=true is not supported for this implementation"
        )

The same attribute-access pattern (request.<field>) appears in all catalog impls: glue.py, hive2.py, hive3.py, unity.py, iceberg.py, polaris.py.

Reproduction (isolated)

from lance_namespace_impls.glue import GlueNamespace
class DictWithModelDump(dict):
    def model_dump(self): return dict(self)
ns = GlueNamespace.__new__(GlueNamespace)
ns.describe_table(DictWithModelDump({"id": ["default", "t"], "load_detailed_metadata": True}))
# -> AttributeError: 'DictWithModelDump' object has no attribute 'load_detailed_metadata'

End-to-end, this surfaces during a Ray → Glue write as:

OSError: LanceError(IO): Failed to fetch storage options: ... Python error in describe_table:
AttributeError: 'DictWithModelDump' object has no attribute 'load_detailed_metadata',
src/namespace.rs:1571 ; lance-io/src/object_store/storage_options.rs

Suggested fix

Make the impls accept a dict-style request (the shape pylance's Rust bridge actually passes), e.g. coerce at the top of each request-taking method:

from lance_namespace_urllib3_client.models import DescribeTableRequest

def describe_table(self, request) -> DescribeTableResponse:
    if not isinstance(request, DescribeTableRequest):
        request = DescribeTableRequest.from_dict(dict(request))
    ...

(Verified working as a monkeypatch shim against the environment above — coercing DictWithModelDump → the real Pydantic model makes attribute access succeed and describe_table returns the correct storage_options.) Alternatively, pylance's call_py_method could pass a real model instead of DictWithModelDump.

The other five impls (hive2, hive3, unity, iceberg, polaris) need the same treatment for their request-taking methods.

Interim workaround

A startup shim (installed via a .pth so it runs in every process) that wraps the impl methods and coerces dict → model before delegating. Happy to open a PR with the in-method coercion if that direction is preferred.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the request-taking methods in glue.py, hive2.py, hive3.py, unity.py, iceberg.py, and polaris.py, then reproduce the isolated DictWithModelDump failure described in the issue. Compare these methods with the wrappers in lance/namespace.py and verify that each catalog implementation handles the Rust bridge request and completes the describe_table write path without AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.