pix2struct + vilt / visual-question-answering: all models pass wmk perf

Summary

All pix2struct and vilt models fail on visual-question-answering. Pix2struct's ONNX config does not list visual-question-answering as a supported task. ViLT has no OnnxConfig registered at all.

Eval Results (2026-03-11)

4/4 models fail — 0 pass.

Root Cause

1. pix2struct: Optimum's pix2struct OnnxConfig only supports image-to-text — not visual-question-answering. Pix2struct performs VQA by taking image + question as input and generating an answer (seq2seq). The task needs to be registered as an alias or a dedicated config added.
2. vilt: No OnnxConfig exists for vilt model type in either Optimum or ModelKit. ViLT (Vision-and-Language Transformer) is a multimodal model requiring custom ONNX export config.

Current State

• No pix2struct.py or vilt.py in modelkit/models/hf/
• Optimum supports pix2struct for image-to-text but not visual-question-answering
• ViLT has no Optimum ONNX support

Desired State

All 4 models pass wmk perf.

Acceptance Criteria

• google/pix2struct-docvqa-base, pix2struct-ai2d-base, google/deplot pass wmk perf
• dandelin/vilt-b32-finetuned-vqa passes wmk perf
• Fix is universal — no hardcoded model IDs (CLAUDE.md Cardinal Rule #1)
• uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

• pix2struct VQA: visual-question-answering is architecturally equivalent to image-to-text for pix2struct (image + text prompt → text answer). Register visual-question-answering as a task alias in a custom Pix2StructIOConfig using @register_onnx_overwrite("pix2struct", "visual-question-answering", ...)
• pix2struct inputs: Uses flattened_patches (not pixel_values) + attention_mask — dummy input generator must handle this non-standard input name
• vilt: ViLT takes pixel_values + input_ids + token_type_ids + attention_mask. Will need a custom ViltOnnxConfig. Check if Optimum has any vilt support upstream.
• Both pix2struct and vilt may also hit the EncoderDecoderCache issue if they use seq2seq generation (pix2struct does)

Related Files

• modelkit/models/hf/blip.py — pattern for multimodal ONNX config registration
• modelkit/export/io.py — register_onnx_overwrite()
• eval_results/2026-03-11/models/google__pix2struct-docvqa-base__visual-question-answering/result.json
• eval_results/2026-03-11/models/dandelin__vilt-b32-finetuned-vqa__visual-question-answering/result.json

Start by reading modelkit/models/hf/blip.py for the multimodal ONNX configuration pattern and modelkit/export/io.py for register_onnx_overwrite(). Review the listed evaluation result files, then run the visual-question-answering checks and uv run pytest tests/. Done means all four named models pass wmk perf without hardcoded IDs and the full test suite passes.