OvertureMaps / OvertureMaps/schema
PySpark codegen silently falls back to StringType for unknown scalars
Open
@sethfitz is already working on this.
Since Jul 28, 2026.
- Dominant language
- Python
- Stars
- 213
- Forks
- 22
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 31
Description
Summary
Types whose validation is supplied by pydantic-core are only covered when the
codegen explicitly implements them. Anything else extracts as a Primitive with
an unregistered base_type, silently falls back to an unchecked StringType(),
and gets zero checks. This also counts for pydantic-extra-types , Overture schema
and 3rd party scalars.
Affected fields today:
sources[].update_time: datetime | None- annex
inception_date: date | None
These are admittedly multiple issues:
- No transparency: the fallback is invisible (logging)
- In-use scalars are missing (datetime/date)
- Most pydantic ecosystem's (core & extra) scalars are missing
- No registration mechanism for third-party scalars
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.