Improvement roadmap: moving every dimension to definitive lead

## Overview

A brutal self-audit of every dimension this benchmark should own. Every item below is something we are currently average, thin, or behind on, relative to where a definitive AI security reasoning benchmark should sit. The goal of this issue is to move every single category from "good enough" to "clearly ahead" with no ties left on the table.

Priority is not ranked inside categories. Categories are grouped by theme.

---

## 1. Thesis and framing

- [ ] Put the one-sentence thesis ("Can your AI reason from raw evidence or just parse labels?") at the very top of the README, in bold, before any statistics or architecture sections
- [ ] Promote the score-discrimination table (regex parser vs frontier LLM across tiers) to hero block position — first table a reader sees
- [ ] Add an explicit "Fully OpenEnv core conformant: `create_app`, `Environment` base class, standard `Action`/`Observation`/`State` contracts" callout
- [ ] Add an explicit "Supports concurrent training sessions — `max_concurrent_envs=4` out of the box" callout
- [ ] Add a short "why this is different" bullet list near the top (raw-evidence reasoning, procedural scenarios, KB-driven, compliance-aware)
- [ ] Add a "limitations" section in the README, honest about current gaps
- [ ] Add a CITATION.cff file for academic reuse

## 2. Scenario generation

- [ ] Add public / holdout split per difficulty (`easy_public`, `easy_holdout`, ...) so training agents cannot overfit to the shipped seeds
- [ ] Demonstrate procedural generation in the README with two worked examples of different seeds producing different scenarios
- [ ] Add scenario version tagging in every emitted scenario so results are reproducible across KB updates
- [ ] Stack-aware vulnerability placement: Django hosts should expose Django-shaped vulns, Spring hosts Spring-shaped vulns, Rails hosts Rails-shaped vulns
- [ ] Expand seed-based generation so the number of distinct scenario shapes is genuinely large, not just the 3 built-in tiers with parameter jitter
- [ ] Gateway-conditioned hidden host unlocking that models a real pivot, not just an `hidden_until` flag
- [ ] Optional seasonal / time-of-day variation that shifts which services are live or silent
- [ ] Deterministic replay: saving a scenario snapshot should let a grader re-run it offline

## 3. Knowledge base depth

- [ ] Expand from 26 vulnerability types to 80+, covering the full OWASP Top 10 2021 exhaustively
- [ ] Add OWASP LLM Top 10 coverage (prompt injection, training data poisoning, insecure output handling, model DoS)
- [ ] Grow payload sets from 16 to 40+ with realistic variations per vulnerability type
- [ ] Raw HTTP transcripts for every vulnerability type in every difficulty tier, not just the hard tier
- [ ] Cross-reference real CVE IDs per vulnerability type so evidence can quote real identifiers
- [ ] Add MITRE ATT&CK technique mappings alongside OWASP / CWE
- [ ] Add CVSS v4.0 support alongside v3.1
- [ ] Add CAPEC attack pattern references where applicable

## 4. Live CVE grounding

- [ ] Optional live NVD / CVE feed integration so scenarios can use current CVEs instead of frozen knowledge
- [ ] Version-aware vulnerability matching: a service version should look up applicable CVEs dynamically
- [ ] Offline cache of CVE data for reproducibility
- [ ] CVE-to-CWE inference for cases where CVE data lacks a direct CWE

## 5. World that pushes back

- [ ] WAF / IDS noise layer: roughly 30 percent of security-control events should be false positives that the agent must filter out
- [ ] State evolution during a session: running a loud scan should trigger IDS alerts that change subsequent tool output
- [ ] Payload-specific responses: the exact payload should influence the response, not just the tool name (currently same template regardless of `" OR 1=1"` vs `"DROP TABLE"`)
- [ ] Alert storm / alert fatigue modeling where signal is buried in noise
- [ ] Simulated rate limiting and blocking under high probe intensity
- [ ] Temporal delay on some indicators (lagging signals — churn-like for security)
- [ ] Race conditions that only surface under specific probe sequences
- [ ] Honeypots that go dormant after one touch, rewarding recovery behaviour

## 6. Authentication and session complexity

- [ ] Multi-role testing with admin / user / guest tokens
- [ ] JWT manipulation test surface (alg=none, key confusion, weak secrets, expired token replay)
- [ ] OAuth 2.0 flow testing (auth code, implicit, PKCE)
- [ ] Session fixation and refresh-token handling
- [ ] Horizontal and vertical IDOR tests grounded in role boundaries
- [ ] MFA bypass scenarios

## 7. Network and topology complexity

- [ ] VLANs and trust zones, not just flat subnets
- [ ] Cloud VPC modeling with security groups and NACLs
- [ ] Shadow IT hosts that only appear when agents deviate from the obvious path
- [ ] Service mesh dependency graph, not just a host list
- [ ] Network blast radius calculation included in findings
- [ ] East-west vs north-south distinction in reachability

## 8. Tool integration and export formats

- [ ] SARIF export of findings for CI / CD integration
- [ ] SBOM / CycloneDX export
- [ ] MCP server interface so findings and tools are consumable by MCP-aware clients
- [ ] Optional sandboxed real-tool wrapping for nmap, nuclei, sqlmap so industry users can validate the parsing layer against real output
- [ ] OpenAPI documentation endpoint (`/openapi.json`) advertised clearly
- [ ] Prometheus `/metrics` endpoint
- [ ] GitHub Advanced Security compatible format
- [ ] JUnit XML export for test-style CI pipelines

## 9. Agent feedback loop

- [ ] Counterfactual coaching messages in grader output: when a dimension scores low, return a string explaining what was wrong and what to try
- [ ] Per-finding explanation of why it matched or did not match ground truth
- [ ] Per-dimension coaching strings on every grade call
- [ ] Agent retry / improvement loop where the agent can submit, receive coaching, and iterate within an episode
- [ ] Episode-level training-signal breakdown exported as JSON for RL post-training consumption

## 10. Grader integrity and anti-gameability

- [ ] Remove or harden the narrative-quality dimension — currently keyword-based and game-able
- [ ] Add gameability tests that assert no amount of keyword stuffing can exceed a score threshold
- [ ] Stricter CWE matching against the full taxonomy, not just exact-ID match (allow parent-CWE credit at reduced weight)
- [ ] Semantic match option for free-form fields via embedding similarity
- [ ] Bayesian finding-to-ground-truth matching to handle imperfect titles
- [ ] Differential grading: publish what an adversarial keyword-stuffing agent scores, and show the gap
- [ ] Inter-rater reliability check: compare grader output against human expert scoring on a sample

## 11. Metrics and reporting

- [ ] Surface the reasoning gap as a first-class metric in every episode report, not only in the README baseline table
- [ ] Time-to-first-true-positive metric
- [ ] False positive rate as a standalone dimension, separate from the penalty
- [ ] Efficiency metric: findings per step
- [ ] Coverage efficiency: unique hosts discovered per scan call
- [ ] Step-level reward trajectory export for learning-curve analysis
- [ ] Confidence intervals on published baseline scores, not point estimates
- [ ] Ablation table showing what each difficulty tier contributes to the reasoning gap

## 12. Baselines and publication

- [ ] Publish baselines across at least five models: regex parser, Llama 3.3 70B, Gemini 2.5 Flash, GPT-4 class, Claude Opus 4.6
- [ ] Publish reasoning gap per model, ranked
- [ ] Publish any RL training curve if one exists
- [ ] Leaderboard on the HuggingFace Space page
- [ ] Reproducibility manifest: exact model versions, seeds, commands, and expected score ranges
- [ ] Human pentester baseline on a sample of scenarios for credibility anchoring

## 13. API surface

- [ ] OpenAPI documentation clearly linked in the README
- [ ] `/metrics` Prometheus endpoint
- [ ] `/version` endpoint returning KB version, scenario version, grader version
- [ ] `/reasoning_gap` endpoint that returns the current run's gap in one call
- [ ] Server-Sent Events streaming for long episodes
- [ ] WebSocket option for interactive agent testing
- [ ] `/export/sarif` endpoint
- [ ] `/export/sbom` endpoint

## 14. Code hygiene

- [ ] Delete the legacy 779-line `server/tools.py` — no file in the codebase imports it
- [ ] Delete the committed `openenv_security_audit_env.egg-info/` directory
- [ ] Resolve the version drift between `pyproject.toml` (`0.1.0`) and `openenv.yaml` (`1.0.0`) — pick one and use it everywhere
- [ ] Add a `LICENSE` file matching the copyright headers currently in source files, or replace headers with our own copyright
- [ ] Remove the contradiction of `uv.lock` being in `.gitignore` while also being tracked
- [ ] Remove the `try / except ImportError` fallback import shims in `app.py`, `security_audit_env_environment.py`, `grader.py`
- [ ] Pick one import style (absolute with `PYTHONPATH=.`) and enforce it everywhere
- [ ] Consolidate duplicated baseline logic between `inference.py` and the `/baseline` endpoint in `app.py`
- [ ] Replace private-attribute access (`env._discovered_hosts`) in `/baseline` with a public accessor

## 15. Tooling and CI

- [ ] GitHub Actions workflow running `pytest`, `ruff`, `mypy`, docker build on every push
- [ ] `ruff` configuration in `pyproject.toml`
- [ ] `mypy` configuration in `pyproject.toml`
- [ ] Pre-commit hooks (ruff, black, mypy, trailing whitespace)
- [ ] Branch protection on main with required checks
- [ ] Semantic versioning and tagged releases
- [ ] Auto-generated CHANGELOG from conventional commits

## 16. Testing depth

- [ ] Property-based tests via Hypothesis on the scenario generator
- [ ] Gameability tests asserting no keyword-stuffing agent can exceed a score threshold
- [ ] Determinism tests: same seed must produce byte-identical output across runs
- [ ] Backwards compatibility tests across scenario KB versions
- [ ] Load test at 100 concurrent sessions to prove the `max_concurrent_envs` claim
- [ ] Optional integration test that runs a real LLM against a fixed seed and asserts a score range
- [ ] Fuzz tests on the action schema validation layer
- [ ] Snapshot tests on tool output to catch unintended KB changes

## 17. Documentation

- [ ] `CONTRIBUTING.md` with development workflow and style guide
- [ ] `CHANGELOG.md` following keep-a-changelog
- [ ] `ARCHITECTURE.md` with flow diagrams for KB → generator → tool engine → grader
- [ ] "How to add a new vulnerability type" walkthrough
- [ ] "How to add a new scenario" walkthrough
- [ ] Agent authoring guide for RL post-training runs
- [ ] Draft technical report or preprint outlining methodology and reasoning-gap thesis
- [ ] FAQ anticipating "is this a real pentest?" / "why not real infra?" / "how do I use this for training?"

## 18. Distribution

- [ ] PyPI package release for the client library
- [ ] Docker Hub pre-built image with tags per release
- [ ] HuggingFace dataset mirror of scenario exports for offline analysis
- [ ] Leaderboard page for community submissions
- [ ] Badge collection in README (tests, coverage, PyPI, docker pulls)

## 19. Niche deepening and expansion tiers

- [ ] OWASP LLM Top 10 prompt-injection scenarios as a new category
- [ ] Mobile app audit tier (Android / iOS static + dynamic testing patterns)
- [ ] IoT / embedded audit tier
- [ ] Cloud misconfiguration tier (IAM, S3, GCS, Azure Blob, RDS exposure)
- [ ] Supply chain audit tier (dependency confusion, typosquat, postinstall hooks)
- [ ] Container security tier (image scanning, K8s RBAC, secrets in manifests)

## 20. Frontier difficulty

- [ ] Adversarial tier: ground truth includes chaff data specifically designed to fool pattern matchers
- [ ] Unknown-vuln tier: the correct answer can be "suspicious but not in the KB" — tests epistemic humility
- [ ] Partial-information tier: the initial scan returns incomplete data and the agent must reason under uncertainty
- [ ] Time-pressure tier: tool fidelity degrades as step count rises
- [ ] Multi-target tier: two networks, one of them a decoy, agent must allocate budget

## 21. Performance

- [ ] Benchmark tool execution latency (p50 / p95 / p99) and publish
- [ ] Optimize scenario generation for 1000+ parallel rollouts
- [ ] Scenario caching strategy for deterministic seeds
- [ ] Async tool execution for independent tools inside one step
- [ ] Memory footprint profile per concurrent session

## 22. Scientific rigor

- [ ] Statistical significance testing on reported baseline gaps
- [ ] Confidence intervals on every published score
- [ ] Ablation study: what does each grader dimension actually contribute to final score variance
- [ ] Sensitivity analysis: how much does seed choice affect score
- [ ] Inter-rater reliability vs human experts on a subsample
- [ ] Calibration curve: predicted agent ranking vs observed ranking

---

## Definition of done

This issue is done when:
1. Every item above is either completed or explicitly marked out of scope with justification
2. A published technical report accompanies the benchmark
3. Baseline scores across at least five models are reproducible by any third party
4. The reasoning gap metric is independently verifiable on scenarios not used in development
5. No category in a public comparison could reasonably be described as "average" or "behind"