Bug: checkpoint validation disappears under optimized Python

Affected area

Model resolution or TensorRT engine build.

TensorRT-Model-Connect version

main at 49aa14fa91d269a37ba8e6d60950aac050e6a2d5.

Installation method

Source checkout. The reproduction is CPU-only and depends on Python optimization semantics, not TensorRT or GPU execution.

Model and revision

Not checkpoint-specific. A source/AST audit found 85 checkpoint and model-config validation assertions across 57 files owned by 55 model families.

Affected families

albert, bert, bloom, codegen, convbert, deberta, deepseek_ocr, deepseek_v2, distilbert, dpr, eagle_vlm, electra, falcon, fnet, gemma, glm, gpt2, gpt_neo, gpt_neox, gpt_oss, granite, internlm, internvl, lance, llama, locateanything, magpie_tts, mamba, mistral, mixtral, modernbert, mpnet, nemotron, nemotron_h, nemotron_labs_diffusion, nemotron_speech_streaming, nemotron_voicechat, olmo, olmo2, opt, personaplex, phi, phi4_multimodal, phi_moe, qwen3_5, qwen3_omni, qwen_moe, qwen_vl, roberta, rwkv, sana_wm, stablelm, starcoder2, xglm, and xlnet.

Qwen is not in this list because #1036 already replaced its removable embedding guard with an explicit ValueError.

Environment

• OS: Windows
• Python: 3.13
• GPU: not required
• CUDA/TensorRT: not required

Minimal reproducer

The affected source pattern is visible with:

git grep -n -E '^[[:space:]]*assert[[:space:]].*(shape|ndim|total_qkv|layer_types|vocab|hidden|heads|size)' -- python/tensorrt_model_connect/families
python -O -c "assert False, 'checkpoint validation'; print('validation skipped')"

A synthetic Llama safetensors control using the exact main loader produced:

# Normal Python
rejected Embedding shape (64, 16) != (32, 16)

# Python -O
accepted_embedding_shape (64, 16)

The synthetic checkpoint declared vocab_size=32 and hidden_size=16 but carried an embedding with shape (64, 16). No model download, TensorRT import, or GPU was required.

Observed behavior

Family-owned checkpoint loaders use assert for externally supplied embedding, position-embedding, token-type, fused projection, layer-pattern, and related tensor/config validation. Python removes those statements when optimization is enabled, so malformed checkpoint data can pass the intended validation boundary and continue into weight mapping or engine construction.

The audit found 122 production assertions in model-family code. This issue is limited to the 85 assertions that validate checkpoint or checkpoint-derived data. It excludes builder/runtime type-narrowing invariants and the separate diffusion bundle-magic checks.

Only two current family tests explicitly expect AssertionError, and there is no repository guard preventing new checkpoint validation from being implemented with assert.

Expected behavior

Externally supplied checkpoint and checkpoint-derived validation must behave identically under normal and optimized Python. Invalid data should raise an explicit, descriptive exception before weight mapping or engine construction continues.

Exit criteria

• Replace all 85 identified checkpoint-boundary assertions with explicit validation that remains active under python -O.
• Use ValueError for malformed checkpoint/config content and preserve existing diagnostic context; add descriptive messages where the assertion currently has none.
• Update the DistilBERT and Nemotron-H tests that currently require AssertionError.
• Add a source-level regression guard that rejects future assert statements in the defined checkpoint-loading boundary.
• Add representative subprocess coverage proving malformed checkpoint data is rejected under python -O.
• Keep the 33 audited builder/runtime internal invariants and the four diffusion bundle-magic assertions out of scope.
• Run focused family tests, repository consistency validation, and public CPU validation without claiming GPU inference, parity, performance, or target qualification.

Submission checks

• I searched open and closed issues and found no duplicate.
• This is not a security vulnerability.
• I removed secrets, credentials, private URLs, and restricted artifacts.

Start with the repository grep command in the issue to enumerate the 85 checkpoint-boundary assertions across python/tensorrt_model_connect/families. Read the affected family loaders and the DistilBERT and Nemotron tests that currently expect AssertionError, then run the synthetic loader under normal Python and python -O. Done means invalid checkpoint data is rejected in both modes, with the source guard and representative subprocess coverage passing.