Consider pinning public-skill context in benchmark metadata

Context

During a local SWE-bench smoke evaluation on an HPC/Apptainer setup, I temporarily added an environment variable to disable load_public_skills() so the run only reflected the benchmark prompt, tools, model checkpoint, and dataset instance.

That local change was not kept for upstreaming because public skills are part of the current OpenHands benchmark harness and are standardized through OpenHands/extensions.

Reproducibility question

For controlled model comparisons, it may still be useful to make the skill set explicit in run metadata, because public skills are loaded from external extension content rather than being fully described by the benchmark repo commit alone. If the extension ref changes, two runs using the same benchmark commit/model/dataset could differ in available skill context.

Possible directions

• Record the resolved EXTENSIONS_REF and loaded skill names in benchmark metadata.
• Add documentation recommending a pinned EXTENSIONS_REF for benchmark papers or checkpoint comparisons.
• Consider an explicit opt-out only for ablation/control runs, if maintainers think that is appropriate.

Non-goal

This is not proposing to disable public skills by default. The issue is only to decide whether benchmark metadata/config should make the loaded public skill context more explicit for reproducibility.

Start by locating the benchmark metadata/configuration and the path that calls load_public_skills(), then trace how EXTENSIONS_REF and loaded skill names are resolved. Review the existing benchmark documentation and metadata outputs before choosing among recording the ref and skills, documenting a pinned ref, or supporting an opt-out. Done means the reproducibility decision is implemented and reflected in metadata or documentation.