blip + blip-2 / visual-question-answering: all models pass wmk perf

Summary

All BLIP and BLIP-2 models fail on visual-question-answering. BLIP reports the task is not supported by its ONNX config. BLIP-2 has no OnnxConfig registered at all for visual-question-answering. ModelKit has a modelkit/models/hf/blip.py but it does not cover the VQA task.

Eval Results (2026-03-11)

4/4 models fail — 0 pass.

Root Cause

1. BLIP (blip): modelkit/models/hf/blip.py registers ONNX configs for image-to-text and image-text-to-text only. visual-question-answering is architecturally equivalent to image-text-to-text for BLIP (image + question → answer) but is not registered as a supported task alias.
2. BLIP-2 (blip-2): No OnnxConfig registered for blip-2 model type at all — modelkit/models/hf/blip.py only covers blip, not blip-2. BLIP-2 uses a different architecture (Q-Former + frozen LLM backbone).

Current State

• modelkit/models/hf/blip.py — exists, covers blip model_type for image-to-text and image-text-to-text
• No registration for blip + visual-question-answering
• No registration for blip-2 model type at all

Desired State

All 4 BLIP/BLIP-2 VQA models pass wmk perf.

Acceptance Criteria

• Salesforce/blip-vqa-base passes wmk perf
• Salesforce/blip2-opt-2.7b passes wmk perf
• Salesforce/blip2-opt-2.7b-coco passes wmk perf
• Salesforce/blip2-flan-t5-xl passes wmk perf
• Fix is universal — no hardcoded model IDs (CLAUDE.md Cardinal Rule #1)
• uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

• BLIP VQA: Add visual-question-answering as a registered task for the blip ONNX config, mapping to the same config as image-text-to-text
• BLIP-2: Complex multi-stage architecture (ViT image encoder → Q-Former → language model). May require a custom Blip2OnnxConfig or separate encoder/decoder export strategy. Check Optimum's Blip2OnnxConfig if it exists.
• BLIP-2 models are large (2.7B params) — may also need external data format for ONNX export
• Salesforce/blip2-flan-t5-xl uses T5 as LLM backbone — the T5 encoder-decoder cache issue (#426) may also apply

Related Files

• modelkit/models/hf/blip.py — existing BLIP config (needs VQA task + blip-2 extension)
• modelkit/models/hf/__init__.py — where blip-2 config import would be added
• modelkit/export/io.py — register_onnx_overwrite()
• eval_results/2026-03-11/models/Salesforce__blip-vqa-base__visual-question-answering/result.json
• eval_results/2026-03-11/models/Salesforce__blip2-opt-2.7b__visual-question-answering/result.json

Start with modelkit/models/hf/blip.py and modelkit/models/hf/init.py, then inspect modelkit/export/io.py and Optimum's Blip2OnnxConfig support. Compare the listed eval result files and existing tests before determining how BLIP and BLIP-2 VQA exports are registered. Done means all four models pass wmk perf and uv run pytest tests/ passes without hardcoded model IDs.