dotnet / dotnet/arcade-services

Automated Known Issue Creation

Open
#6,584 0 comments 0 reactions 0 assignees View on GitHub
ai-drafted build-insights
Dominant language
C#
Stars
86
Forks
86
Avg merge
1d 8h
Merged PRs (30d)
35

Description

## Business value

Reduce manual known-issue authoring and repeated CI triage, while making automated creation, matching evidence and its impact visible.

## Expected outcomes

Unmatched failures produce validated known issues under an explicitly chosen review policy:

- **Approval-first:** candidate rule -> back-testing -> human approval -> creation.
- **Create-first:** candidate rule -> back-testing -> tracking-only creation -> highlight for human review -> confirmation enables excusing.
- **Full auto (open decision):** back-testing -> creation and activation without user interaction, potentially from day one.

## Acceptance criteria

- [ ] Eligible unmatched/partially matched failures produce candidate descriptions and matching rules without duplicate entries from repeated processing.
- [ ] Back-testing reports originating/related matches, unrelated non-matches and likely false positives. Manual authors can test rules against selected builds before saving.
- [ ] Create-first entries track/report matches without excusing failures. Users are informed of newly auto-matched issues and can confirm they are safe. Review-required modes do not excuse unconfirmed issues.
- [ ] Human confirmation applies to the known issue for current and future matching builds. The Build Insights check turns green only when every blocking failure is covered by an issue authorized to excuse under the chosen policy, with no other blockers remaining.
- [ ] Every automatically created entry visibly identifies its AI authorship. Origin survives human approval; creation counts/shares distinguish manual from automated authoring. Creator, reviewers, decisions and activation history remain auditable.
- [ ] Creation, review and match-testing respect existing data/repository permissions. Rejected issues cannot excuse failures.

## Scope

**In:** Generation, validation/manual testing, review, activation and provenance/reporting as specified above.

**Out:** Fixing underlying failures, changing retry behavior or overriding unrelated CI checks.

## Dependencies and prerequisites

Confirm failure evidence, known-issue creation/matching capabilities, permissions and representative evaluation builds. Related UI/platform work is not automatically a blocker.

## Risks and mitigations

Over-broad rules could hide genuine regressions. Assess false-positive impact versus UX gains on representative cases; agree controls for erroneous activation and revocation.

## Delivery assessment

> Delivery assessment is **experimental**; estimates and AI-suitability ratings are provisional.

- **Size:** `XL` (5+ engineering days); refine and consider decomposition after prioritization.
- **Complexity drivers:** Generation, back-testing, review/activation, notifications and provenance metrics.
- **AI implementation suitability:** `Medium` - coding assistance is useful; automation policy and evaluation need human decisions.
- **Human checkpoints:** Agree mode, risk tolerance, evaluation cases, permissions and activation/revocation behavior.
- **Assessment assumptions:** Scope-based estimate; implementation coverage and historical effort are unreviewed. Re-estimate after mode/coverage decisions.

## Related

- [dotnet/arcade-services#6590](https://github.com/dotnet/arcade-services/issues/6590): MVP/go-live context.
- [dotnet/build-insights#61](https://github.com/dotnet/build-insights/issues/61) and [dotnet/build-insights#62](https://github.com/dotnet/build-insights/issues/62): known-issue list/statistics and manual-authoring UI.
- [dotnet/build-insights#77](https://github.com/dotnet/build-insights/issues/77): Data API; consider validation/provenance metrics for non-UI consumers.

The Build Insights links are historical; successors remain unverified. Legacy closure means cleanup, not delivery.

## Open decisions

- **Launch policy:** Select the initial mode and the evidence/risk thresholds for enabling full-auto.
- **Validation:** Eligibility, partial-match/retry handling, evaluation windows and acceptable false-positive impact.
- **Operations:** Reviewers, notification surfaces, rejection disposition, rule changes and revocation/rollback.

---
🤖 _Drafted by an AI agent on behalf of @michalpavelka._

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the related arcade-services issue #6590 and the historical Build Insights issues #61, #62, and #77, then confirm the available failure evidence, known-issue matching, permissions, and representative evaluation builds. Done requires an agreed launch policy and validated coverage for generation, back-testing, review, activation, provenance, permissions, and rejection or revocation behavior.

Written by the indexing model from the issue text.

Assessment

Domain
ci-cd, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.