OvertureMaps / OvertureMaps/schema
[ENHANCEMENT](codegen) Decide support-or-reject for Pydantic Field kwargs the extractor ignores or crashes on
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 213
- Forks
- 22
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 31
Description
The codegen extractor reads a narrow slice of each field's Field() metadata: description, metadata (the annotated_types / pydantic constraint objects), alias / validation_alias, default / default_factory (presence only, for is_required), annotation, and the union-level discriminator (model_extraction.py:40-53,160-179; type_analyzer.py:303-306). Any other Field() kwarg falls into one of two unsignposted failure modes: it is silently dropped (documentation kwargs that live as FieldInfo attributes the extractor never reads) or it hard-crashes make generate-pyspark (validation kwargs that land in field_info.metadata, reach dispatch_constraint, and hit the unhandled-constraint raise).
This issue enumerates the specific kwargs to make a decision on. For each: either support it, or reject it with a legible error instead of the current silence-or-stacktrace. Current behavior below is verified against HEAD.
Silently dropped
These are FieldInfo attributes, not field_info.metadata entries, so they never reach constraint dispatch and never raise. The extractor simply never looks at them, and nothing tells the schema author that the kwarg had no effect.
title
Field(title=...) is never read. The value vanishes; the generated Markdown reference shows nothing for it.
Open question, and the reason this needs design rather than a one-line attribute read: what should a field title render as in the Markdown reference? We already surface description; a title is a shorter human label that may or may not belong in the output at all, and if it does, its placement (heading, bolded lead-in, a column) is undecided. This item is a rendering decision first, an extraction change second.
examples (aspirational)
Field(examples=...) is ignored. Doc examples come exclusively from each theme's pyproject.toml [examples.<model>], validated against the model (extraction/examples.py). Letting a field carry its own inline example via Field(examples=) -- as a supplement to, or lighter-weight alternative to, the pyproject source -- is a nice-to-have, not a current gap. Lower priority than the rest; listed so the decision is explicit rather than implicit.
Crashes pyspark generation
These land in field_info.metadata as _PydanticGeneralMetadata, are collected unfiltered by the extractor, and reach dispatch_constraint, which has no handler for them and raises TypeError: Unhandled constraint type: _PydanticGeneralMetadata (constraint_dispatch.py:410). No caller wraps that call, so the first field to use any of these aborts make generate-pyspark. The Markdown path does not raise -- it falls through to a repr() fallback and emits the raw _PydanticGeneralMetadata(...) string into the docs (field_constraints.py:118-120) -- so docs and pyspark disagree on what is even recognized.
Nothing in the schema uses these today, which is the only reason the build is green. They are latent: the first use breaks it loudly (pyspark) and silently-wrong (docs) at the same time.
allow_inf_nan
Field(allow_inf_nan=False) -> TypeError on the pyspark path (verified live). Decision: translate to a Spark check (reject inf/NaN on the column) or reject the kwarg at extraction with a message naming it.
max_digits
Field(max_digits=...), a Decimal constraint -> same TypeError. No schema field is currently Decimal. Decision folds together with decimal_places and with whether Decimal fields are in scope at all.
decimal_places
Field(decimal_places=...), a Decimal constraint -> same TypeError. Pairs with max_digits.
Supported, but with a cliff: pattern
Field(pattern=...) is supported on both paths, so this is not an unsupported-kwarg entry -- it is a partial-support gotcha worth capturing here rather than discovering at build time.
A plain string pattern (Field(pattern="...")) carries no flags and works. But a compiled pattern with any regex flag beyond IGNORECASE / UNICODE -- re.MULTILINE, re.DOTALL, re.VERBOSE, re.ASCII -- raises NotImplementedError, naming the flag, rather than mis-translating it to a Spark rlike that would silently match differently (constraint_dispatch.py:145-165, gate at :163). That raise is correct behavior, not a bug.
The gotcha is the asymmetry: the Markdown path renders those same flags as an inline (?ms...) group without raising (field_constraints.py:143-155), so a flagged pattern can render fine in the docs and then abort pyspark generation. A schema author adding a multiline pattern sees green docs and a broken build. Worth either aligning the two paths (docs also refuse what pyspark refuses) or documenting the supported-flag set at the point patterns are authored. Field-level Field(pattern="...") cannot carry flags, so this only bites when a schema constraint class compiles its own flagged pattern.
Out of scope here
deprecated is tracked separately in #639. Other FieldInfo kwargs are also dropped -- frozen, exclude, serialization_alias, json_schema_extra -- but are not part of this issue's decision set; split out separately if any warrant support.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with model_extraction.py:40-53,160-179 and type_analyzer.py:303-306 to trace Field metadata into constraint_dispatch.py:410; compare the Markdown handling in field_constraints.py:118-120,143-155. Run make generate-pyspark with representative kwargs and review extraction/examples.py for existing example behavior. Done means each in-scope kwarg has an explicit support or legible rejection decision, with docs and PySpark behavior aligned where applicable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100