microsoft / microsoft/SkillOpt

[Request] Release optimized skill artifacts for additional methods / models / harnesses

Open
#130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.3k
Forks
1.6k
Avg merge
2d 7h
Merged PRs (30d)
17

Description

Hi SkillOpt team,

Thank you for open-sourcing the best GPT-5.5 SkillOpt checkpoints under ckpt/
(https://github.com/microsoft/SkillOpt/tree/main/ckpt) — they have been very
helpful for getting started.

Context

I'm doing an analysis of the learned skills and would like to build on SkillOpt.
Since optimizing these skills is token-expensive and I'm compute/budget-constrained,
re-running the full loop to reproduce every cell in Table 1 is impractical on my end.
Because the exported artifacts are compact best_skill.md files (per the paper, mostly
<2,000 tokens), releasing them should be low-cost on your side and would let the
community study and compare the skills directly, and improve reproducibility.

Request

Would it be possible to release the exported skill artifacts (the final best_skill.md,
and ideally the accepted-edit history / selection & test scores if available) for the
following cells?

1. Direct-chat artifacts — one skill per benchmark
(SearchQA, SpreadsheetBench, OfficeQA, DocVQA, LiveMath, ALFWorld):

Method GPT-5.5 GPT-5.4 GPT-5.4-mini GPT-5.4-nano GPT-5.2 Qwen3.5-4B Qwen3.6-35B-A3B
SkillOpt ✅ done
Trace2Skill
GEPA

2. Harness artifacts (GPT-5.5) — 5 benchmarks, no ALFWorld:

Method Codex harness Claude Code harness
SkillOpt
EvoSkill

Note: I limited the EvoSkill request to the Codex / Claude Code harnesses because,
if I read Table 1 correctly, EvoSkill is only reported there and not in the
direct-chat rows. Please correct me if direct-chat EvoSkill artifacts also exist.

Priority (in case a full release is too much at once)

If releasing everything is impractical, an incremental release in this order would
already be very useful to me:

  1. SkillOpt across the remaining 6 direct-chat models (the headline artifacts).
  2. GEPA and Trace2Skill across all 7 direct-chat models (for cross-method comparison).
  3. The Codex / Claude Code harness artifacts (SkillOpt + EvoSkill).

Happy to help however useful — e.g. verifying/organizing the released files, or writing
a short README for the extra checkpoints — and I will of course cite the paper.

Thanks again for the great work and for open-sourcing it!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The currently released artifacts are in ckpt/ as compact best_skill.md files; start by reviewing those files and the Table 1 cells listed in the issue. Done means publishing the requested additional artifacts and, where available, their accepted-edit histories and selection/test scores.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.