feat: explore replay-guided Context Policy optimization

背景与调整

Dream-RSI 的核心思想是把已发生的探索历史组织成 replay simulator，先低成本比较探索策略，再把有希望的策略放回真实环境验证。

PowerContext 不会依赖Dream-RSI 的代码；所有 contract、record、Runtime 边界、评估流程和安全控制都由 PowerContext 独立设计和实现。

我们不再把 prepare_context 作为第一落点。它是每次准备上下文都会调用的延迟敏感路径，选择规则通常已受业务配置、Scope 和预算约束。它更适合作为 approved Experience 的下游消费者和效果观测点。

PowerContext 更自然的自进化边界是 Experience / Skill：bounded task evidence 形成 Experience proposal；approved Experience、Source 或 usage evidence 形成或修订 managed Skill；人工 Review 后得到 immutable Revision；后续任务再产生真实 usage outcome。

因此本 Issue 包含两个相关提议：

1. 调整自进化的作用边界： 把自进化能力放在 Experience / Skill generation，而不是 prepare_context。
2. 丰富现有 Dream 阶段： PowerContext 在这个生成环节已有一个相对简单的 Dream 阶段；在保留 Review 和 Artifact 生命周期的前提下，为它增加多尝试、评估、提名、下游反馈和历史回放能力。

两者的关系是：Experience / Skill generation 是“优化什么”的边界，Dream 是“如何进行探索和自我改进”的机制。下文的 Artifact Evolution Policy 只是 Dream 内部的薄编排边界，不是一套脱离现有流程的通用 policy platform。

核心提议

引入内部 Artifact Evolution Policy，显式编排 evidence grouping、目标选择、attempt budget、prune/stop 和 nomination：

bounded exact evidence
  -> Artifact Evolution Policy
  -> isolated Proposal Attempts
  -> deterministic / model evaluation
  -> nominate at most one pending ArtifactCandidate
  -> human Review
  -> approved immutable Revision
  -> downstream Experience recall or Skill usage
  -> task / usage outcome
  -> evolution replay
  -> improve the orchestration policy

必须区分：

• Proposal Attempt 是 evaluation-owned 探索记录，可以有多个、失败或被剪枝；它不是 Artifact，也不进入 Review Inbox。
• ArtifactCandidate 仍是 Review Inbox 中的正式对象。每个 evolution objective 最多提名一个 pending Candidate。

Dreaming 不能批准 Candidate、发布/安装/执行 Skill、授予权限或修改 active Revision。

该层直接复用PowerContext 现有约束：ArtifactGenerationInput 最多 32 条 exact evidence、每条最多 64,000 字符；reviewed generation 返回完整 typed proposal 或 no-op；Skill origin 已区分 experience、source、usage；所有 create/revise 仍经过 Candidate、Review、Revision 和 lineage；Skill usage evidence 已包含 exact Skill ref、package digest、target、selected、invoked、validation、outcome、task source 和 environment fingerprint。

如何丰富现有 Dream

Dream 不是 Experience / Skill generation 的同义词，而是这个生成过程中的探索与提炼阶段。现有实现中，它最直接地体现在自动 Experience incubation：

bounded Task Outcome window
  -> fixed incubation prompt
  -> one generation call
  -> 0..N Experience proposals
  -> every validated proposal becomes a pending Candidate

此issue不是简单增加 prompt，而是从两个层次丰富它：

1. 对象层 Dream（P2/P3）：改进生成什么。 先把一个 Source window 划分成一个或多个独立 evolution objectives；每个 objective 在硬预算内执行不同 evidence grouping 的 Proposal Attempts；评估 grounding、lineage、重复、冲突、target 和成本；最后 no-op 或最多提名一个 Candidate。一个 window 可以有多个互不相关的 objectives，因此仍可能产生多个 Candidates，但每个 Candidate 都来自独立、可解释的提名过程。
2. 元策略层 Dreaming（P4）：改进以后如何生成。 把 P2/P3 真实发生过的 attempt、评估、Review 和 downstream outcome 组成 realized evolution trees，在其中比较 attempt allocation、prune/continue/stop 和 nomination policy，再把 replay winner 放回 fresh live run 验证。

现有：evidence -> one-shot generation -> Candidates

丰富后：
evidence
  -> objectives
  -> isolated multi-attempt generation
  -> evaluate / prune / continue / stop
  -> nominate
  -> Review and immutable Revision
  -> downstream outcome
  -> replay and improve the evolution policy

P0 只形式化当前行为，P1 只增加 faithful recording/replay；它们不宣称提高生成质量。实际功能扩展从 P2 开始。P3 再把闭环扩展到 Experience-to-Skill 和 usage-driven replacement。论文中的 replay-based “dreaming”主要对应 P4，而不是把每次在线 generation 都称为 replay。

Replay 边界

Replay 只能比较实际发生过的分支。只有 exact evidence、target、generator/prompt/schema identity 和参数匹配的 historical attempt 才能被揭示和重新计分。未执行过的 evidence combination 必须返回 out_of_support，不能推断它会生成什么，也不能获得虚构 reward。

所以 replay 用于比较已实现分支上的 attempt 顺序、预算、剪枝、停止和提名，不是生成模型的离线 world model。任何 replay winner 都必须经过新的 live evolution run。

P0–P4

P0：Baseline policy boundary

• 把当前 Experience incubation、reviewed generation、Experience-to-Skill 和 usage-driven evolution 描述为 baseline policy。
• 保持现有 API、Candidate 数量语义、模型调用、lineage 和 Review 行为不变。
• 行为测试证明引入内部边界前后 observable result 相同；无 public API 或 migration。

现有 incubation 的多输出在 P0 仅被忠实记录；“每个 objective 最多一个 Candidate”从新的 P2 exploration path 开始执行。

P1：Evolution Run 与 faithful replay

• 增加 evaluation-owned Evolution Run / Proposal Attempt record，保存 exact snapshot、policy/generator identity、action、output/no-op/error、evaluation、cost 和 nomination。
• Replay 读取保存的 attempt result，不重新调用模型；不支持的 action fail closed 为 out_of_support。
• 普通生产流量默认不保存完整 evidence body，仓库 fixture 只用 synthetic/redacted evidence。

P2：Experience-first bounded exploration

• 从 task-outcome evidence 开始，在固定 attempt/token/wall-time/concurrency budget 内探索多个 evidence grouping 或 generation path。
• 先检查 schema、exact citation、Scope、grounding、duplicate、conflict 和 target correctness。
• 每个 objective 最多一个 Experience proposal 进入 Review Inbox；development/holdout 分离，再做 fresh live validation。

P3：Experience-to-Skill 与 usage-driven evolution

• 先探索 approved Experience 到 managed Skill，再探索 exact Skill Revision + bounded usage evidence 的 replacement。
• selected、invoked、validation、outcome 保持独立，unknown/non-invoked 不能包装成成功。
• 用 exact Revision、package digest、target 和 environment fingerprint 分层比较 task quality、runtime、tokens、turns 和 tool calls。

P4：Replay-guided policy improvement

• 用 P2/P3 的 realized trees 比较 repository-owned 或 bounded declarative policy。
• Policy 只能看到已揭示 observation，不能读 hidden outcome、holdout label 或未来分支。
• Replay 改善只是 nomination evidence；policy 变更仍需 fresh live validation、人工代码/配置 Review 和显式部署。
• 第一阶段不执行模型生成的任意 policy code。

评估与成本

不能只优化 Review approval rate。建议分别报告：

• 即时有效性：schema、exact evidence、grounding、lineage、Scope、target、conflict、fabricated outcome；
• Review：approved/revised/rejected 和 reason；
• 延迟效果：Experience recall/usefulness，Skill selected/invoked/validated，task outcome、runtime、tokens、turns、model/tool calls；
• 演化成本：attempts、generation/evaluator tokens、wall time、concurrency、failure rate、Review time 和 amortization。

延迟的真实任务结果应是 promotion 的主要证据。多分支探索会增加演化 job 的时间和成本，因此默认只在异步、显式或 evaluation workflow 中运行，不能进入 prepare_context 热路径。Baseline 与 candidate 必须使用相同硬预算，并同时报告下游收益和全部探索成本。

希望讨论

1. Artifact Evolution Policy 是否是迁移该机制的正确首个边界？
2. P2 是否应严格从 Experience generation 开始，再进入 Skill？
3. Evolution Run / Proposal Attempt 应长期只属于 evaluation artifact，还是需要受控 persistence？
4. 第一批 action 应只包含 evidence grouping、attempt budget、prune/stop 和 nomination，还是包含 target/split？
5. 哪些 delayed signals 足以支持 promotion，如何处理跨任务、Skill 和环境的 credit assignment？
6. 第一轮 workload、预算、holdout 和 promotion gates 应如何预注册？

English

Background and revised direction

The key idea in Dream-RSI is to organize realized exploration history as a replay simulator, compare exploration policies cheaply, and return promising policies to the real environment for validation.

PowerContext will not depend on the any Dream-RSI code. PowerContext independently defines every contract, record, Runtime boundary, evaluation workflow, and safety control.

prepare_context is no longer the first target. It is a latency-sensitive path invoked whenever context is prepared, and its choices are usually constrained by business configuration, Scope, and budgets. It is better treated as a downstream consumer of approved Experience and an observation point for effectiveness.

Experience / Skill is the more natural evolution boundary in PowerContext: bounded task evidence produces Experience proposals; approved Experience, Sources, or usage evidence produce or revise managed Skills; human Review creates immutable Revisions; and later tasks produce real usage outcomes.

This issue contains two related proposals:

1. Move the self-evolution boundary: apply self-evolution to Experience / Skill generation rather than prepare_context.
2. Enrich the existing Dream stage: PowerContext already has a relatively simple Dream stage in this generation lifecycle. Add multiple attempts, evaluation, nomination, downstream feedback, and historical replay while preserving Review and the Artifact lifecycle.

Their relationship is: Experience / Skill generation defines what is improved, while Dream defines how exploration and self-improvement are organized. Artifact Evolution Policy below is only a thin internal orchestration boundary inside Dream, not a generic policy platform detached from current flows.

Core proposal

Introduce an internal Artifact Evolution Policy that orchestrates evidence grouping, objective selection, attempt budgets, prune/stop decisions, and nomination:

bounded exact evidence
  -> Artifact Evolution Policy
  -> isolated Proposal Attempts
  -> deterministic / model evaluation
  -> nominate at most one pending ArtifactCandidate
  -> human Review
  -> approved immutable Revision
  -> downstream Experience recall or Skill usage
  -> task / usage outcome
  -> evolution replay
  -> improve the orchestration policy

The distinction is important:

• A Proposal Attempt is an evaluation-owned exploration record. Many attempts may fail or be pruned. It is not an Artifact and never enters the Review Inbox.
• An ArtifactCandidate remains a formal Review Inbox object. At most one pending Candidate may be nominated per evolution objective.

Dreaming cannot approve a Candidate, publish/install/execute a Skill, grant authority, or mutate an active Revision.

This layer reuses current PowerContext constraints: ArtifactGenerationInput accepts at most 32 exact evidence items of at most 64,000 characters each; reviewed generation returns a complete typed proposal or no-op; Skill origins distinguish experience, source, and usage; every create/revise still passes Candidate, Review, Revision, and lineage; and Skill usage evidence already records exact Skill ref, package digest, target, selected, invoked, validation, outcome, task source, and environment fingerprint.

How the current Dream stage is enriched

Dream is not a synonym for Experience / Skill generation. It is the exploration and distillation stage within that generation process. In the current implementation, its clearest automated form is Experience incubation:

bounded Task Outcome window
  -> fixed incubation prompt
  -> one generation call
  -> 0..N Experience proposals
  -> every validated proposal becomes a pending Candidate

The proposal enriches this stage at two levels instead of merely expanding the prompt:

1. Object-level Dream (P2/P3): improve what is generated. Split one Source window into one or more independent evolution objectives. For each objective, run Proposal Attempts over different evidence groupings within hard budgets; evaluate grounding, lineage, duplication, conflict, target correctness, and cost; then return no-op or nominate at most one Candidate. A window may contain multiple unrelated objectives and therefore still produce multiple Candidates, but each Candidate comes from an independent, explainable nomination process.
2. Meta-policy Dreaming (P4): improve how future generation is organized. Build realized evolution trees from actual P2/P3 attempts, evaluations, Review decisions, and downstream outcomes. Compare attempt-allocation, prune/continue/stop, and nomination policies in those trees, then validate a replay winner in a fresh live run.

current: evidence -> one-shot generation -> Candidates

enriched:
evidence
  -> objectives
  -> isolated multi-attempt generation
  -> evaluate / prune / continue / stop
  -> nominate
  -> Review and immutable Revision
  -> downstream outcome
  -> replay and improve the evolution policy

P0 only formalizes current behavior, and P1 only adds faithful recording/replay; neither claims better generation quality. Functional enrichment starts in P2. P3 extends the loop to Experience-to-Skill and usage-driven replacement. The paper's replay-based "dreaming" maps primarily to P4, rather than making every online generation call a replay operation.

Replay boundary

Replay can compare only branches that were actually realized. A historical attempt may be revealed and rescored only when exact evidence, target, generator/prompt/schema identity, and parameters match. An unexecuted evidence combination is out_of_support: replay cannot infer its output or assign fabricated reward.

Replay therefore compares attempt order, budgets, pruning, stopping, and nomination over realized branches. It is not an offline world model for the generator. Every replay winner requires a fresh live evolution run.

P0-P4

P0: baseline policy boundary

• Describe current Experience incubation, reviewed generation, Experience-to-Skill, and usage-driven evolution as a baseline policy.
• Preserve current APIs, Candidate-count semantics, model calls, lineage, and Review behavior.
• Prove equivalent observable results with behavior tests; add no public API or migration.

P0 only records the current multi-output incubation behavior faithfully. The at-most-one-Candidate rule begins with the new P2 exploration path.

P1: Evolution Run and faithful replay

• Add evaluation-owned Evolution Run / Proposal Attempt records containing exact snapshots, policy/generator identity, action, output/no-op/error, evaluation, cost, and nomination.
• Replay saved attempt results rather than calling a model again. Unsupported actions fail closed as out_of_support.
• Ordinary production traffic does not retain full evidence bodies by default; repository fixtures use synthetic/redacted evidence.

P2: Experience-first bounded exploration

• Start with task-outcome evidence and explore multiple evidence groupings or generation paths under fixed attempt/token/wall-time/concurrency budgets.
• Gate schema, exact citation, Scope, grounding, duplication, conflict, and target correctness first.
• At most one Experience proposal per objective enters Review. Separate development/holdout data and perform fresh live validation.

P3: Experience-to-Skill and usage-driven evolution

• First explore approved Experience to managed Skill, then replacement from an exact Skill Revision plus bounded usage evidence.
• Preserve selected, invoked, validation, and outcome separately. Unknown/non-invoked observations cannot become success.
• Stratify task quality, runtime, tokens, turns, and tool calls by exact Revision, package digest, target, and environment fingerprint.

P4: replay-guided policy improvement

• Compare repository-owned or bounded declarative policies over realized P2/P3 trees.
• Policies observe revealed state only, never hidden outcomes, holdout labels, or future branches.
• Replay improvement is nomination evidence only. Policy changes still require fresh live validation, human code/configuration Review, and explicit deployment.
• The first stage does not execute arbitrary model-generated policy code.

Evaluation and cost

Review approval rate cannot be the sole objective. Report separately:

• immediate validity: schema, exact evidence, grounding, lineage, Scope, target, conflict, and fabricated outcome;
• Review: approved/revised/rejected and reason;
• delayed effect: Experience recall/usefulness, Skill selected/invoked/validated, task outcome, runtime, tokens, turns, and model/tool calls;
• evolution cost: attempts, generation/evaluator tokens, wall time, concurrency, failure rate, Review time, and amortization.

Delayed real-task outcomes should dominate promotion. Multi-branch exploration increases evolution-job time and cost, so it runs only in asynchronous, explicit, or evaluation workflows by default and never on the prepare_context hot path. Baseline and candidate receive equal hard budgets, with both downstream benefit and total exploration cost reported.

Questions

1. Is Artifact Evolution Policy the right first boundary for adapting this mechanism?
2. Should P2 start strictly with Experience generation before moving to Skill?
3. Should Evolution Run / Proposal Attempt remain evaluation artifacts, or eventually receive controlled persistence?
4. Should initial actions cover only evidence grouping, attempt budgets, prune/stop, and nomination, or also target/split decisions?
5. Which delayed signals support promotion, and how should credit be assigned across tasks, Skills, and environments?
6. How should the first workload, budgets, holdout, and promotion gates be preregistered?

References

• Dream-RSI paper
• 中文解读 / Chinese overview

No files or tests are named. Start by tracing the existing Experience incubation, reviewed generation, Experience-to-Skill, and usage-driven evolution entry points, then follow Candidate, Review, Revision, and lineage handling. Done would require an agreed scope and validation plan for the proposed P0–P4 evolution and replay stages without changing the prepare_context hot path.