Policy on Figure-Based Questions in `courseexam` Benchmark

We currently disallow any questions that reference figures or images in the courseexam benchmark (see the no-images/figures policy
), even though many modern LLMs are natively multimodal.

The main motivation for this policy is to avoid penalizing text-only models, which still make up a large portion of the models we evaluate. In addition, the goal of the benchmark is to assess models’ ability to solve systems programming exam questions. We want to ensure that differences in image-handling capabilities do not influence the results or confound comparisons between models.

One alternative that was discussed is to tag questions that include images and exclude them when comparing multimodal and text-only models. However, this would complicate scoring, since total exam points are used to compare model performance against real student results when those are available.

For now, we have decided to only include questions that do not reference images or figures. We are retaining the original exam PDFs so that we can revisit this decision in the future if we choose to support image-based questions, either through tagging or via a separate benchmark.

This issue is intended to track this design decision and the ongoing discussion.

Related discussion: https://github.com/sys-intelligence/system-intelligence-benchmark/pull/61#discussion_r2688669429

Read the no-images/figures policy in benchmarks/courseexam_bench and the related discussion on PR #61. Review the retained original exam PDFs and confirm that this issue records the current decision and rationale, while leaving future image-support approaches for later discussion.