epam / epam/ai-dial-admin-evaluation-framework-backend

[Eval] [tech] Establish recurring retrospective and self-improvement loop for agent and SDD rules/processes

Open
#30 0 comments 0 reactions 2 assignees Claimed by @Katerina-Charakhovich View on GitHub
enhancement
Dominant language
Java
Stars
3
Forks
1
Avg merge
1d 1h
Merged PRs (30d)
27

Description

### Name and Version

0.2.0

### What is the problem this feature will solve?

The project relies on a growing set of agent-facing rules and spec-driven-development (SDD) processes — agent guidance documents, OpenSpec configuration and workflow, SDD slash commands and skills, and agent-tooling settings (hooks, permissions, environment). These artifacts have accumulated organically alongside feature work and currently work well, but there is no scheduled moment to step back and ask:

- Are the rules still consistent with each other and with how the codebase has evolved?
- Which rules are actively enforced vs. quietly ignored?
- Where does the SDD workflow add real value, and where is it ceremony?
- Are slash commands, skills, and hooks pulling their weight, or are they stale?
- What recurring friction or surprises have agents and humans hit since the last review?

Without a deliberate retrospective cadence, this guidance tends to drift: rules contradict each other, conventions documented in one place stop matching another, and improvements that "everyone agrees on" never land because no one owns the cleanup. This ticket exists so that retrospective and self-improvement on these rules/processes is not forgotten — there is no acute issue to fix today.

### What is the feature you are proposing to solve the problem?

Establish a recurring, lightweight **retrospective and self-improvement loop** covering all agent- and SDD-facing rules and processes as one umbrella effort. In scope:

- Agent-facing repository guidance (top-level agent/contributor docs and per-tool memory).
- OpenSpec workflow — config, artifact rules, archive checklist, spec index, and the spec set itself.
- SDD slash commands and skills (the `opsx:*` / `openspec-*` family and any project-specific skills).
- Agent-tooling settings — hooks, permissions, environment variables, and similar harness configuration.

High-level behavior of the loop:

- **Cadence**: a fixed recurring slot (e.g. monthly or per-milestone) at which a short retrospective is run; out-of-band reviews are allowed when a sharp pain point appears.
- **Inputs**: recently archived changes, friction observed during agent sessions, contradictions or stale references spotted in passing, and proposals queued since the last round.
- **Outputs**: a small, prioritized list of concrete edits — rule clarifications, deletions of dead guidance, new lint/CI checks, slash-command tweaks, hook adjustments — landed as one or more focused changes through the normal SDD workflow.
- **Ownership**: a named owner per round who runs the retro, drafts the action list, and ensures follow-ups are filed; rotates over time.
- **Self-application**: the loop itself is subject to the same retro — its cadence, scope, and format are tunable based on whether it is producing useful improvements.

The goal is an explicit, low-overhead habit of reflection rather than a heavyweight governance process. No code-behavior changes, no public API changes, no schema changes.

### What alternatives have you considered?

- **Ad-hoc fixes only** — address rule/process issues opportunistically when they bite. Rejected: this is the current de-facto state; it works for sharp pain but lets quiet drift accumulate and discourages broader simplifications.
- **One-time audit + cleanup PR** — schedule a single deep review and call it done. Rejected: rules and processes evolve continuously alongside the codebase; a one-shot review goes stale within a release or two.
- **Automated enforcement only** (lint rules, CI checks for spec/code consistency, hook-driven reminders) — without a retrospective layer. Rejected: automation is a useful *output* of the retro, but cannot decide which rules deserve to exist, which should be relaxed, or which slash commands have outlived their purpose.
- **Per-area reviews in isolation** (separate retros for agent docs, OpenSpec, skills, hooks) — Rejected: many of the friction points sit at the seams between these areas (e.g. a rule in agent guidance that contradicts an OpenSpec config setting); a single umbrella loop catches cross-cutting issues.

### Additional context

This is an internal tech-debt / process item, not a user-facing feature. Success looks like: a steady stream of small, deliberate improvements to agent and SDD guidance over time, fewer cross-document contradictions, and the loop itself getting lighter or heavier as evidence dictates.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.