OvertureMaps / OvertureMaps/schema
AnyScalar falls back to StringType silently; codegen should raise instead
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 213
- Forks
- 22
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 31
Description
Summary
overture-schema-codegen falls back to representing fields as strings in several cases, and not all of them are correct. #591 covers two of those cases; this is a third, and it needs a different fix.
Concretely, _spark_for_scalar reaches _STRING_FALLBACK by four paths:
# codegen/pyspark/schema_builder.py
def _spark_for_scalar(scalar: Scalar) -> str:
if isinstance(scalar, (LiteralScalar, AnyScalar)): # (1) explicit branch
return _STRING_FALLBACK
if scalar.base_type in SHARED_TYPE_REFS: ...
if enum_source(scalar) is not None: # (2) deliberate, enums are strings
return _STRING_FALLBACK
return _spark_for_base(...) # (3,4) registry misses -> #591
Paths 3 and 4 are a scalar the registry does not know, and a registration mechanism fixes them. Path 1 is not a lookup failure: AnyScalar is the IR's faithful record that the model said Any. There is no type to register and no lookup that could have succeeded.
The proposal
Codegen should raise on AnyScalar rather than silently substitute a target type.
Targets differ in what they can honestly do with Any. Some have a genuine equivalent — an untyped object, a variant type — and can render it faithfully. Others have nothing that loose, and there the renderer is choosing a constraint on the model's behalf. StringType() is that second case: it states something the model does not, and states it invisibly, since the emitted schema looks authored.
So the decision belongs at the model, not in each renderer's default. Raising forces it into the open: either the model narrows the type, or it carries an explicit annotation saying how to project it where the target cannot express it. Either way the choice is recorded somewhere a reader can find it, which a per-renderer fallback never is.
This subsumes #591's first sub-issue (transparency/logging) for this path: with a raise, there is nothing to log.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in codegen/pyspark/schema_builder.py at _spark_for_scalar and inspect the explicit LiteralScalar/AnyScalar branch and the existing fallback behavior. Change the AnyScalar path to raise instead of returning _STRING_FALLBACK, then run the relevant codegen tests and add coverage showing that AnyScalar no longer emits StringType.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 75/100