internlm2 + phi4mm / visual-question-answering: all models pass wmk perf

Summary

internlm/internlm-xcomposer2d5-7b and microsoft/Phi-4-multimodal-instruct fail because they contain custom model code that requires trust_remote_code=True. ModelKit does not currently pass this flag when loading these models.

Eval Results (2026-03-11)

2/2 models fail — both require trust_remote_code=True.

Root Cause

Both models define custom Python code in their HuggingFace repository (custom modeling files, processors, etc.) that must be executed to load the model. HuggingFace requires trust_remote_code=True as an explicit opt-in for security reasons. ModelKit currently does not expose or pass this flag.

Current State

• modelkit/loader/config.py:21-109 — WinMLLoaderConfig has a trust_remote_code field but it may default to False or not be passed through to the HF loader
• modelkit/loader/hf.py — HF model loading; check if trust_remote_code is forwarded
• wmk perf CLI — no --trust-remote-code flag exposed

Desired State

Both models pass wmk perf when run with trust_remote_code=True.

Acceptance Criteria

• wmk perf -m internlm/internlm-xcomposer2d5-7b --trust-remote-code passes
• wmk perf -m microsoft/Phi-4-multimodal-instruct --trust-remote-code passes
• --trust-remote-code flag exposed in wmk perf and wmk build CLI commands
• trust_remote_code properly propagated through WinMLLoaderConfig → HF loader
• Fix is universal — applies to any model requiring trust_remote_code (CLAUDE.md Cardinal Rule #1)
• uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

• WinMLLoaderConfig.trust_remote_code exists (modelkit/loader/config.py) — verify it is forwarded to AutoModel.from_pretrained(trust_remote_code=...) in modelkit/loader/hf.py
• Add --trust-remote-code / -T flag to wmk perf and wmk build CLI commands
• Security note: this flag should never be set by default — always explicit opt-in
• Note: even with trust_remote_code=True, these are large multimodal LLMs (7B params) — may also hit ONNX size limits or timeout. Address trust_remote_code first, then size/timeout separately if needed.

Related Files

• modelkit/loader/config.py:21-109 — WinMLLoaderConfig with trust_remote_code field
• modelkit/loader/hf.py — HF model loading; AutoModel.from_pretrained() call
• modelkit/commands/perf.py — CLI command (needs --trust-remote-code option)
• modelkit/commands/build.py — CLI command (needs --trust-remote-code option)
• eval_results/2026-03-11/models/microsoft__Phi-4-multimodal-instruct__visual-question-answering/result.json

Review modelkit/loader/config.py, modelkit/loader/hf.py, and the CLI entry points in modelkit/commands/perf.py and build.py. Trace how WinMLLoaderConfig reaches the HuggingFace loading call, then run the stated model commands and relevant tests; done means explicit --trust-remote-code support works for both commands without enabling it by default and the test suite passes.