OvertureMaps / OvertureMaps/schema

Code generation mishandles NamedTuple fields

Open
#667 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
213
Forks
22
Avg merge
3d 3h
Merged PRs (30d)
31

Description

Found while scoping #568.

typing.NamedTuple and collections.namedtuple are both accepted by Pydantic as field annotations, and validated eagerly, field by field. The code generator extracts them as a Primitive whose unregistered base_type degrades to an unchecked StringType() — the same silent failure mode as #568 and #591, but with a different correct answer, so it is filed separately.

No Overture model uses a NamedTuple today.

Current behavior

class Point(NamedTuple):
    x: int
    y: str

class Holder(BaseModel):
    location: Point
Annotation FieldShape Spark schema type Markdown
Point (NamedTuple) Primitive('Point') StringType() (wrong) Point (wrong)
the same shape as a BaseModel ModelRef StructType([StructField("x", LongType(), True), StructField("y", StringType(), True)]) (correct) PointModel (correct)

An untyped collections.namedtuple("Plain", "a b") is accepted too, with Any-typed fields, and degrades identically.

Why this is not #568

The collection types in #568 — set, tuple[X, ...], Sequence, Mapping — all materialize into a homogeneous sequence or mapping, so ArrayOf / MapOf is the right shape and the only question is how to reach it.

A NamedTuple has named fields with heterogeneous types. That is a struct. A Spark ArrayType is homogeneous, so the array route is not physically expressible for Point above.

The decision this issue needs

Pydantic serializes a NamedTuple positionally, as an array:

>>> Holder(location=Point(1, "a")).model_dump(mode="json")
{'location': [1, 'a']}

and its JSON Schema agrees:

{"type": "array", "minItems": 2, "maxItems": 2,
 "prefixItems": [{"title": "X", "type": "integer"}, {"title": "Y", "type": "string"}]}

So each candidate mapping costs something:

  • StructType with named fields matches the Python type's semantics, and matches what an equivalent BaseModel already generates — but disagrees with model_dump(), which emits an array.
  • Follow Pydantic and emit an array keeps the generated schema aligned with serialization — but is inexpressible in Spark for heterogeneous field types, and loses the field names.
  • Refuse NamedTuple annotations with an error pointing at BaseModel, which is what a schema author almost certainly wants. Given that no model uses one today, this is the cheapest correct answer, and the one I would default to.

Interaction with #568

issubclass(Point, tuple) is True and get_origin(Point) is None, so a NamedTuple reaches _terminal in type_analyzer.py rather than any container branch.

The fix for #568 is likely to add a bare-tuple guard there, mirroring the existing ones:

if issubclass(annotation, list):
    raise TypeError("Bare list without type argument is not supported")

A NamedTuple would be swept into that guard and fail with Bare tuple without type argument is not supported, which does not describe the problem. Whoever implements #568 should either handle NamedTuple explicitly or make sure it does not land in that branch.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The relevant entry point is _terminal in type_analyzer.py; begin by tracing how NamedTuple annotations reach it and compare the interaction with #568. Decide which of the three mappings the project will support, then add coverage for typed and untyped NamedTuple cases and verify the generated Spark schema and Markdown output match that decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.