OvertureMaps / OvertureMaps/schema
Code generation mishandles NamedTuple fields
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 213
- Forks
- 22
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 31
Description
Found while scoping #568.
typing.NamedTuple and collections.namedtuple are both accepted by Pydantic as field annotations, and validated eagerly, field by field. The code generator extracts them as a Primitive whose unregistered base_type degrades to an unchecked StringType() — the same silent failure mode as #568 and #591, but with a different correct answer, so it is filed separately.
No Overture model uses a NamedTuple today.
Current behavior
class Point(NamedTuple):
x: int
y: str
class Holder(BaseModel):
location: Point
| Annotation | FieldShape | Spark schema type | Markdown |
|---|---|---|---|
Point (NamedTuple) |
Primitive('Point') |
StringType() (wrong) |
Point (wrong) |
the same shape as a BaseModel |
ModelRef |
StructType([StructField("x", LongType(), True), StructField("y", StringType(), True)]) (correct) |
PointModel (correct) |
An untyped collections.namedtuple("Plain", "a b") is accepted too, with Any-typed fields, and degrades identically.
Why this is not #568
The collection types in #568 — set, tuple[X, ...], Sequence, Mapping — all materialize into a homogeneous sequence or mapping, so ArrayOf / MapOf is the right shape and the only question is how to reach it.
A NamedTuple has named fields with heterogeneous types. That is a struct. A Spark ArrayType is homogeneous, so the array route is not physically expressible for Point above.
The decision this issue needs
Pydantic serializes a NamedTuple positionally, as an array:
>>> Holder(location=Point(1, "a")).model_dump(mode="json")
{'location': [1, 'a']}
and its JSON Schema agrees:
{"type": "array", "minItems": 2, "maxItems": 2,
"prefixItems": [{"title": "X", "type": "integer"}, {"title": "Y", "type": "string"}]}
So each candidate mapping costs something:
StructTypewith named fields matches the Python type's semantics, and matches what an equivalentBaseModelalready generates — but disagrees withmodel_dump(), which emits an array.- Follow Pydantic and emit an array keeps the generated schema aligned with serialization — but is inexpressible in Spark for heterogeneous field types, and loses the field names.
- Refuse NamedTuple annotations with an error pointing at
BaseModel, which is what a schema author almost certainly wants. Given that no model uses one today, this is the cheapest correct answer, and the one I would default to.
Interaction with #568
issubclass(Point, tuple) is True and get_origin(Point) is None, so a NamedTuple reaches _terminal in type_analyzer.py rather than any container branch.
The fix for #568 is likely to add a bare-tuple guard there, mirroring the existing ones:
if issubclass(annotation, list):
raise TypeError("Bare list without type argument is not supported")
A NamedTuple would be swept into that guard and fail with Bare tuple without type argument is not supported, which does not describe the problem. Whoever implements #568 should either handle NamedTuple explicitly or make sure it does not land in that branch.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The relevant entry point is _terminal in type_analyzer.py; begin by tracing how NamedTuple annotations reach it and compare the interaction with #568. Decide which of the three mappings the project will support, then add coverage for typed and untyped NamedTuple cases and verify the generated Spark schema and Markdown output match that decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100