LFX Mentorship Tracking: Automated Documentation Sync Bot for Krkn-Chaos Projects

Tracking issue for LFX 2026 (Term 2). Implements krkn-chaos/website#320.
Mentee: Ayush Kumar(@StrikerEureka34)
Mentors: Darshan Jain(@ddjain) Paige Patton(@paigerube14), Tullio Sebastiani(tsebastiani), Naga Ravi Chaitanya Elluri(@chaitanyaenr), Rahul Shetty(@rh-rahulshetty)

## Motivation

The krkn-chaos/website repo hosts the unified Hugo/Docsy docs for the whole ecosystem. When an upstream project changes a scenario, a config field, or a CLI flag, someone has to update the docs by hand, which causes documentation drift. This project builds a bot (GitHub Action plus an LLM-assisted agent) that detects documentation-impacting changes upstream and opens draft PRs on the website following existing conventions.

## Expected outcome

- Trigger on documentation-impacting upstream changes (config fields, CLI flags, new scenarios, parameter tables) and open draft PRs on krkn-chaos/website.
- Follow existing Hugo/Docsy structure: tabbed scenario pages, parameter tables rendered from data files via a shortcode.
- Keep extraction deterministic; the LLM writes only parameter descriptions, with a mechanical fallback.
- Produce review-friendly, draft-only PRs (trigger attribution, parameter diff, cost stats), never auto-merge.
- Support interactive commands and refinement (`/fix`, `/resync`, `/refine`) with authorization.
- Detect drift on a schedule and report it as a structured issue with `/fix` commands.
- Scaffold new scenario pages that have no website page yet.
- Ship unit tests and documentation about the bot.
- Scope: Phase 1 krkn, krkn-hub, krknctl; Phase 2 krkn-ai, krkn-operator, krkn-assist.

## Solution overview

- **Deterministic extraction:** regex parser for `env.sh`, JSON parser for `krknctl-input.json`, YAML parser for `krkn/config.yaml`; global params filtered via a skip-list; no LLM in extraction, so names and defaults are exact.
- **LLM descriptions only:** provider-agnostic interface, one sentence per parameter, existing descriptions preserved, mechanical fallback if the LLM fails.
- **Data files plus shortcode (built):** the bot writes `data/params//.yaml` and never edits human markdown; one `param-table` shortcode renders every table with a canonical schema and auto-hide columns.
- **Draft PR generation:** draft-only, with trigger attribution, a parameter diff, cost stats, and hidden `` metadata for `/resync`; branch `docs-bot/`.
- **Drift detection and ChatOps:** weekly scan posts a structured issue with a per-scenario checklist; `/fix`, `/resync`, and `/refine` commands.
- **gh-aw hardening (under evaluation):** read-only agent job, egress firewall, safe-outputs gate, and threat detection; one agentic `/refine` wrapped in a deterministic guard that lets the model touch only `description` fields.

## Security and safety constraints

- **Draft-only PRs, no auto-merge.** Every change is human-reviewed.
- **Authorization:** `author_association` check (MEMBER, COLLABORATOR, OWNER only); scenario validated against a mapping file; rate limited and concurrency-grouped per scenario.
- **Tokens and permissions:** scoped, short-lived tokens (a GitHub App token with 1-hour expiry, not a PAT, or the auto-expiring `GITHUB_TOKEN` if going App-free); `permissions:` blocks at the workflow level; the gh-aw option keeps the write token only in the gated safe-outputs job.
- **Supply chain:** third-party actions SHA-pinned; secret masking in logs (`::add-mask::`).
- **LLM safety:** input sanitization (cap value length, strip control characters) and output validation (reject URLs and shell commands) to resist prompt injection.
- **Build safety:** Hugo build check (`hugo --minify`) before a PR opens; on failure, post a comment instead of a broken PR.
- **Reliability:** concurrency groups keyed on scenario with cancel-in-progress; atomic commits (all data files staged together, committed only on full success); GitHub API retries with exponential backoff.

## Milestones

Two phases, matching the repo scope: Phase 1 is the core sources (krkn, krkn-hub, krknctl), Phase 2 is the expansion repos (krkn-ai, krkn-operator, krkn-assist) plus wrap-up. Boxes are ticked to the week each item actually shipped, so some Phase 1 items completed ahead of their original week.

### Phase 1: Core sources (krkn, krkn-hub, krknctl)

**Week 1 (Jun 8 to 14): Onboarding and audit**
- [x] Complete LFX and Slack setup; meet mentors for the project overview.
- [x] Set up a local Hugo/Docsy build of the website.
- [x] Audit the scenario content structure (tabbed pages, parameter tables, all-scenario-env).
- [x] Study the upstream sources of truth (env.sh, krknctl-input.json).
- [x] Design the param-table data schema and the auto-hide column policy.

**Week 2 (Jun 15 to 21): Shortcode build and gh-aw research report**
- [x] Build the param-table shortcode and write 14 edge-case tests; fix the 2 bugs found.
- [x] Migrate the node-scenarios and service-hijacking tabs; verify output matches production.
- [x] Open the template PR and a typo-fix PR.
- [x] Prepare a research report on retrofitting gh-aw into the existing setup: pros and cons, security model, and costs.
- [x] Present the findings in the weekly sync.

**Week 3 (Jun 22 to 28): Finalize gh-aw scope and identity**
- [x] Write the gh-aw integration design and the two source workflows (doc-sync, doc-refine).
- [x] Retrofit the krkn-hub bot to run under gh-aw and test it.
- [x] If an App is needed, set it up for the demo.
- [x] Sync with mentors to finalize the extent of gh-aw to use and whether a GitHub App is needed.
- [x] Resolve the engine choice (route to an OpenAI-compatible endpoint; Copilot gpt-4o-mini works for now).

**Week 4 (Jun 29 to Jul 5): gh-aw working end to end after the mentor nudge**
- [x] Move the deterministic work (install, clone, parse, generate, commit) out of the agent into gh-aw custom steps after the mentor suggestion; agent cost dropped from ~177k to ~53k tokens per run.
- [x] Deterministic `env.sh` parser to `data/params//krkn-hub.yaml` (names, types, defaults exact).
- [x] krknctl extraction (`krknctl-input.json`) to `krknctl.yaml`, with a description join that fills env-only params from the krknctl descriptions.
- [x] Global-parameter skip-list from `all-scenario-env.md`.
- [x] End-to-end krkn-hub bot: `/fix` and a merged-PR dispatch open a draft PR, with DCO sign-off, running on a cheap engine.
- [x] Security envelope in place via gh-aw: draft-only PRs, a GitHub App short-lived token, `permissions: read-all`, a read-only agent job with an egress firewall, the safe-outputs gate holding the write token, secret masking, and threat detection.
- [x] id-mapping via `` so scenarios whose page directory differs (e.g. `node-cpu-hog` to `hog-scenarios/cpu-hog-scenario`) resolve to the right page.
- [x] New scenario-page scaffolding for scenarios with no website page yet.
- [x] Confirm the scope and feasibility of `/refine` under the no-Copilot and no-App constraints.

**Week 5 (Jul 6 to 12): formalize, batch, and stress test**
- [x] Hugo build check gating every docs PR, with branch protection requiring build + DCO.
- [x] Multi-scenario batching: one source PR that touches several scenarios now produces a single docs PR (fixed the trigger's `head -1` silent-drop bug).
- [x] End-to-end stress test across modify, delete, add, and new-page in one PR ([krkn-hub #45](https://github.com/StrikerEureka34/krkn-hub/pull/45) to [website #43](https://github.com/StrikerEureka34/website_2/pull/43)), written up with a [pass table](https://github.com/krkn-chaos/docsync-bot/issues/5).
- [x] Scan the website scenario naming and directory divergences against upstream, and draft the id-mapping enforcement proposal (the "fix directory structure" groundwork).
- [x] Open formal PRs to the `docsync-bot` repo (bot code and workflow, split by target repo, with per-folder READMEs and TODOs).
- [x] Complete documentation for all the PRs made.

**Week 6 (Jul 13 to 19): third source (krkn), drift and define scope for onboarding**
- [x] krkn `config.yaml` parser (`extract_krkn_config_params`) and wire `krkn` as the third source, emitting `krkn.yaml`; the `param-table` shortcode is already source-agnostic.
- [x] Stand up a krkn fork trigger and test the third source end to end on the fork rig (same pattern as the krkn-hub trigger).
- [x] Verify prompt-injection behaviour (threat detection already runs on every job; deliberately stress-test it with a weaker engine and document the result).
- [x] Wire the weekly drift scan to a schedule and post a structured drift issue with `/fix` commands.
- [x] Fix the website scenario directory and naming structure (the id-mapping enforcement PR).
- [x] Define and discuss the requirements for onboarding the phase-1.

**Week 7 (Jul 20 to 26): Midterm review and demo**
- [x] Midterm evaluation (Jul 21).
- [ ] Demo the full loop on the fork rig: a real krkn-hub change to a data PR, plus `/fix`, `/resync`, a drift report, multi-scenario batching, and new-page scaffolding.
- [x] Collect the review's feedback and suggestions, and start applying them.

**Week 8 (Jul 27 to Aug 2): Apply midterm feedback and onboard Phase 1 to the real repos**
- [x] Finish the midterm review changes and suggestions.
- [ ] Onboard Phase 1 completely to the real repos: register the org-owned GitHub App (#3), add triggers to the real krkn-hub and krkn repos, and target the real krkn-chaos/website (krknctl already flows through the krkn-hub trigger).
- [x] Tighten command authorization from the testing `roles: all` to `[admin, maintainer, write]`, and validate the scenario against a mapping file.
- [ ] Remaining hardening: an explicit per-scenario concurrency group with cancel-in-progress, rate limiting, pin `actions/checkout` to a SHA, and API retries with backoff (atomic commits and secret masking are already in place).
- [ ] Live-test end to end on the real Phase-1 repos (krkn, krkn-hub, krknctl) and the real website.

### Phase 2: Expansion repos (krkn-ai, krkn-operator, krkn-assist) and wrap-up

**Week 9 (Aug 3 to 9): Extend extraction and triggers to the expansion repos**
- [x] Study the doc sources and config formats for krkn-ai, krkn-operator, and krkn-assist.
- [x] Add extraction for their config formats and wire them as new sources (reusing the deterministic parser and the source-agnostic shortcode).
- [ ] Add cross-repo triggers for the three repos.
- [x] Scaffold their scenario or config pages where none exist yet (reusing the existing scaffolding).

**Week 10 (Aug 10 to 16): Validate the expansion repos**
- [x] Test end to end on krkn-ai, krkn-operator, and krkn-assist plus the website.
- [ ] Fold in review feedback and fix any format-specific edge cases.

**Week 11 (Aug 17 to 23): Tests and docs**
- [ ] Complete unit-test coverage across all sources (67 as of 12th July, fill the gaps).
- [ ] Write the documentation about the bot.
- [ ] Resolve remaining review feedback.

**Week 12 (Aug 24 to 31): Wrap-up**
- [ ] Final evaluation (Aug 25).
- [ ] Refinement, clean commits, and a final demo.
- [ ] Handoff documentation for whoever comes next.

## Appendix

- Upstream project issue: krkn-chaos/website#320
- Related issues in this repo: #3 (bot identity and handover), #4 (agent scope reduction findings)
- Sibling project for reference: https://github.com/redhat-community-ai-tools/code-to-docs
- GitHub Agentic Workflows: https://github.github.com/gh-aw/

Start by reviewing the named extraction sources: env.sh, krknctl-input.json, and krkn/config.yaml, then inspect the existing workflows and data/params//.yaml outputs. Check the unit-test coverage and the remaining unchecked milestones, especially real-repository onboarding, hardening, cross-repo triggers, and bot documentation. Done means the remaining Phase 1 and Phase 2 work is implemented, tested, documented, and live-tested.

Written by the indexing model from the issue text.