OvertureMaps / OvertureMaps/schema

PySpark codegen silently falls back to StringType for unknown scalars

Open
#591 0 comments 0 reactions 1 assignee View on GitHub

@sethfitz is already working on this.

Since Jul 28, 2026.

Dominant language
Python
Stars
213
Forks
22
Avg merge
3d 3h
Merged PRs (30d)
31

Description

Summary

Types whose validation is supplied by pydantic-core are only covered when the
codegen explicitly implements them. Anything else extracts as a Primitive with
an unregistered base_type, silently falls back to an unchecked StringType(),
and gets zero checks. This also counts for pydantic-extra-types , Overture schema
and 3rd party scalars.

Affected fields today:

  • sources[].update_time: datetime | None
  • annex inception_date: date | None

These are admittedly multiple issues:

  • No transparency: the fallback is invisible (logging)
  • In-use scalars are missing (datetime/date)
  • Most pydantic ecosystem's (core & extra) scalars are missing
  • No registration mechanism for third-party scalars

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.