siglip / zero-shot-image-classification: all models pass wmk perf

Summary

google/siglip-so400m-patch14-384 fails with an ONNX Runtime optimization error while smaller SigLIP variants pass. This is the only SigLIP model failing — the large so400m variant has a 384px patch size that may produce a graph that ONNX Runtime cannot optimize.

Eval Results (2026-03-11)

1/3 siglip models fails — larger variant only.

Root Cause

siglip-so400m-patch14-384 uses 384×384 input resolution with patch size 14, producing 729 patches (vs 196 for patch16-224). The much larger attention matrix likely triggers an ONNX Runtime graph optimization pass that fails — possibly a shape inference issue or unsupported operation at that scale.

Note: so400m means ~400M params (So ViT-400M) — significantly larger than base variants.

Current State

• modelkit/models/hf/ — no siglip.py (siglip uses clip-like architecture, may share config)
• Smaller siglip variants pass without explicit config
• so400m-patch14-384 is the only failing variant

Desired State

google/siglip-so400m-patch14-384 passes wmk perf.

Acceptance Criteria

• google/siglip-so400m-patch14-384 passes wmk perf
• Existing siglip-base-patch16-224 and marqo-fashionSigLIP continue to pass
• Fix is universal — no hardcoded model names (CLAUDE.md Cardinal Rule #1)
• uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

• The ONNX Runtime error is during the optimization pass (not inference), meaning the graph is too complex or has an unsupported pattern for ORT's optimizer at this input resolution
• First try: disable specific ORT optimizations for this graph (e.g., set optimization level to ORT_ENABLE_BASIC instead of ORT_ENABLE_ALL)
• If an unsupported op is the cause: identify which optimization pass fails and skip it via disabled_optimizers in the session options
• The laion/CLIP-ViT-H-14-laion2B-s32B-b79K (zero-shot-image-classification) also fails with onnx.ModelProto (size issue, tracked in #429) — separate cause

Related Files

• modelkit/models/hf/clip.py — SigLIP shares CLIP-like dual-encoder architecture, may reuse configs
• modelkit/session/ — ONNX Runtime session configuration and optimization level settings
• eval_results/2026-03-11/models/google__siglip-so400m-patch14-384__zero-shot-image-classification/result.json

Reproduce the failure with google/siglip-so400m-patch14-384 using wmk perf, then inspect modelkit/session/ for ONNX Runtime session and optimization settings and modelkit/models/hf/clip.py for the shared model path. Use the existing eval result for the failure context. Done means the large SigLIP model passes without a model-specific workaround, the two passing SigLIP models remain passing, and uv run pytest tests/ succeeds.