Mining is limited to programmatic success checks — intent-level improvements get dropped or Goodharted into shallow proxy rules

ORIGIN & MOTIVATION

I'm exploring using SkillOpt as a step toward "on-the-job training" for AI Employees learning a specific job role from real work sessions. This issue is about what looks like a structural ceiling on that goal in the mining stage. (Related but distinct from #67, which covers reward-hacking on the gate side; this is about the miner side.)

Problem

The task miner only retains a candidate task if success can be expressed as a programmatic check (regex-style assertions). That means behaviors like "understand my intent better" don't compress into a check, so they are either:

1. dropped the most valuable improvement signal never enters the pipeline, or
2. mangled into a shallow proxy a phrasing/format preference observed in one session gets distilled into a hyper-literal rule that is then robotically applied to all future outputs. Goodhart's law in miniature: the check becomes the target.

I've observed the second failure mode in my own runs: a one-off summary phrasing preference became a rigid rule stamped onto every future summary.

If mining is restricted to regex-expressible outcomes, the system can only ever improve the regex-expressible slice of agent behavior, which is a small subset of what users actually repeat-and-rephrase about in real sessions.

Possible directions (heuristics, not designs)

1. Divergence-point detection: mine topic-shift structure in transcripts (user descends into a rabbit hole on a sub-issue → resolves it → conversation returns to the main thread) as natural task boundaries and implicit failure signals, instead of requiring a programmatic outcome check.
2. Clean-context replay comparison: a second instance with fresh context attempts the reconstructed task; a judge compares its output against what the user ultimately accepted in the original transcript, rather than against a regex proxy.
3. Baseline-instance divergence: an instance seeded only with the user's global rules/skills replays the session's user prompts in order until its output clearly and definitely diverges from the transcript — the divergence point marks where a learnable behavior lives, and bounds the task to mine.

Question for maintainers

Is there interest in supporting judge-based (non-programmatic) success checks in mining/replay, and what evidence standard would you consider acceptable for gating on them?

Start by tracing the task miner's regex-based success checks and the mining/replay flow described in the issue. Compare the proposed divergence-point, clean-context replay, and baseline-instance approaches, then review the maintainers' response to determine whether a judge-based evidence standard is accepted; the issue is complete only when a concrete design and acceptance criteria are agreed.