align() silently reinterprets int64 columns as epoch timestamps (int64 → timestamp[us]) instead of raising

Schema.align() promises "safe, no-data-change operations" (PyArrowSchema docstring: "casting, when it can be done safely"), but when a column arrives as int64 and the schema wants timestamp[us], align silently reinterprets the integers as microseconds-since-epoch. Every value lands near 1970-01-01 with zero warnings — silent data corruption from a schema-alignment API.

Repro (flexible_schema 0.1.1, pyarrow 23.0.1, Python 3.12)

import pyarrow as pa
from flexible_schema import PyArrowSchema

class Data(PyArrowSchema):
    subject_id: pa.int64()
    time: pa.timestamp("us")
    code: pa.string()

tbl = pa.Table.from_pydict({
    "subject_id": [1, 1, 2],
    "time": [0, 10, 120],   # intended as minutes-from-admission, NOT epoch microseconds
    "code": ["A", "B", "C"],
})
out = Data.align(tbl)
print(out.column("time").to_pylist())
# [datetime(1970, 1, 1, 0, 0), datetime(1970, 1, 1, 0, 0, 0, 10), datetime(1970, 1, 1, 0, 0, 0, 120)]

No error, no warning. The same behavior is inherited by meds.DataSchema.align (meds 0.4.1), where it corrupts the mandatory time column of MEDS datasets.

Where the cast happens

src/flexible_schema/pyarrow.py:327-328 (v0.1.1; unchanged at current main):

@classmethod
def _cast_raw_table_column(cls, table: pa.Table, col: str, want_type: pa.DataType) -> pa.Table:
    return table.set_column(table.schema.get_field_index(col), col, table.column(col).cast(want_type))

reached from Schema.align → _cast_raw_table (src/flexible_schema/base.py:397 → :351-352), which casts every mistyped column indiscriminately. ChunkedArray.cast defaults to safe=True, but pyarrow's "safe" mode does not help here: int64 → timestamp is defined in Arrow as an epoch-unit reinterpretation and is permitted even with safe=True (verified with both ChunkedArray.cast(..., safe=True) and pc.cast(..., safe=True)). So the permissive decision is effectively flexible_schema's — it needs its own cast policy rather than pyarrow's.

Current contract (empirically, for a timestamp[us] target)

int64→timestamp is the lone pair where align succeeds while changing the meaning of the data. A schema aligner should restrict itself to semantic-preserving casts (widening within a kind, string parsing that can fail loudly) or raise; reinterpreting an integer's unit/epoch is a transformation, not an alignment.

Suggested fix

In PyArrowSchema._cast_raw_table_column (or a shared policy hook in base.py), reject cross-kind reinterpretation casts before delegating to pyarrow — minimally:

if pa.types.is_integer(current_type) and (pa.types.is_timestamp(want_type)
        or pa.types.is_date(want_type) or pa.types.is_time(want_type)):
    raise SchemaValidationError(...)  # report as a mistyped column, like the float64 case

or, more robustly, an explicit allowlist of permitted cast pairs (same-kind widening, string→parsed types, int→float) with everything else surfacing as the existing SchemaValidationError: Columns with incorrect types: .... If someone genuinely has epoch integers, an opt-in escape hatch (e.g. a per-column or per-call allow_reinterpret_casts=True) keeps that workflow available without making it the silent default.

This would make int64 behave like the already-correct float64/int32 cases: a loud SchemaValidationError naming the column and both types.

Context

Found via mmcdermott/MEDS_extract#207: a MEDS extraction config that binds time: to an integer offset column (e.g. eICU's observationoffset, forgetting the pseudotime derivation) sails through the whole pipeline and DataSchema.align at the finalize stage silently lands every event at 1970-01-01.

Start with src/flexible_schema/pyarrow.py:_cast_raw_table_column and trace its callers through src/flexible_schema/base.py:_cast_raw_table and Schema.align. Use the provided int64-to-timestamp reproduction to verify that alignment raises SchemaValidationError naming the column and types instead of silently producing epoch timestamps.