SWE-bench Multimodal: 2 instances always fail due to robots.txt-blocked image URLs

Summary

Two SWE-bench Multimodal instances (markedjs__marked-684 and Automattic__wp-calypso-26008) consistently fail across all agent configurations (OpenHands, delegation, Claude SDK × Opus/Sonnet) because the Anthropic API rejects image URLs that are blocked by robots.txt.

Root Cause

The issue descriptions for these instances contain image URLs hosted on cloud.githubusercontent.com and user-images.githubusercontent.com. The benchmark harness passes these URLs directly to the Anthropic API as image_urls in the message payload. The Anthropic API attempts to fetch the images server-side, but these GitHub CDN domains block external fetching via robots.txt, causing the API to return a 400 Bad Request:

AnthropicException - {"type":"error","error":{"type":"invalid_request_error",
"message":"This URL is disallowed by the website's robots.txt file."}}

The error is fatal — it triggers a ConversationErrorEvent with code LLMBadRequestError before the agent can do any work. All 3 retries fail identically.

Affected Instances

markedjs__marked-684 — 3 images from cloud.githubusercontent.com:

• cloud.githubusercontent.com/assets/87356/5483878/54978340-8639-11e4-961b-7f5f293980c6.jpg
• cloud.githubusercontent.com/assets/87356/5483883/66d6c32c-8639-11e4-87cf-a18d96fd59e8.jpg
• cloud.githubusercontent.com/assets/87356/5483887/7d846408-8639-11e4-9201-a86ded31a5d1.jpg

Automattic__wp-calypso-26008 — 3 images from user-images.githubusercontent.com / cloud.githubusercontent.com:

• user-images.githubusercontent.com/17325/42547087-72d982f8-8514-11e8-8fcb-48a78a6b0401.png
• cloud.githubusercontent.com/assets/7233112/22263721/630de9e2-e243-11e6-8476-b425d8ac5a77.png
• cloud.githubusercontent.com/assets/7233112/22263743/7876fcf6-e243-11e6-931f-62451d55512c.png

Impact

• 2/102 instances (≈2%) are completely broken across all runs
• These instances are counted as failures, slightly deflating benchmark scores

Suggested Fixes

1. Pre-download images and send as base64: Instead of passing image URLs to the API, the harness could download the images at dataset preparation time or at runtime before the API call, and send them as base64-encoded inline images. This would avoid any robots.txt restrictions.
2. Graceful degradation: If image fetching fails, catch the error and retry the API call with the text-only prompt (without images). The agent would lose visual context but could still attempt the task.
3. Dataset-level fix: Re-host the affected images at URLs that don't block API fetching (e.g., store them as part of the benchmark dataset assets).

Reproduction

Run any SWE-bench Multimodal evaluation — these two instances will fail on every attempt with the error above.

Run a SWE-bench Multimodal evaluation and trace how the harness passes image_urls to the Anthropic API for markedjs__marked-684 and Automattic__wp-calypso-26008. Compare the proposed image download, graceful degradation, and dataset re-hosting approaches; done means these instances no longer fail fatally because GitHub image URLs are blocked by robots.txt.