canonical / canonical/test_observer

Support artefact-scoped attachment rules (low priority; easy workaround exists)

Open
#832 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
11

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated triage experience.**
Filed by @canonical/solutions-qa

## Priority
**Low.** There is a straightforward workaround (see below), so this is a usability/ergonomics improvement rather than a blocker.

## Summary
Allow issue-to-test-result attachment rules to be scoped to a specific artefact (charm/snap). Today the attachment-rule create schema (`IssueTestResultAttachmentRulePostRequest`) accepts `families`, `environment_names`, `test_case_names`, `template_ids`, `test_result_statuses`, and `execution_metadata` — but no artefact dimension.

## Problem
When a single charm fails with a **variable** failure signature, there is no way to write one clean rule:
- Keying on `execution_metadata` (a specific endpoint/message) matches only one variant — you'd need N rules for N signatures.
- Keying on `test_case_names` (e.g. `test_build_bundle`) matches that test case for **every** charm, which massively over-attaches.
- Net result: the issue must be attached by explicit result-ID lists, and future recurrences are not auto-attached.

## Workaround (why this is low priority)
The situation is easy to work around today:
- Attach the issue to the affected results directly via an explicit `test_result_ids` list (precise, no over-matching), or
- Create one narrow rule per stable `execution_metadata` signature.

The only real cost is manual re-attachment when new failures of the same variable-signature bug appear.

## Evidence
`self-signed-certificates` bundle-build fails across ~30 different neighbor-endpoint signatures (charm-integration-testing#747). With no artefact-scoped rule available, the issue was manually attached three times (revisions v650 ×2, v652), ~236 explicit result IDs total, with no auto-attach going forward. By contrast, `kubeapi-load-balancer` and `nova-cloud-controller` had a single stable signature, so a normal `execution_metadata` rule worked cleanly.

## Proposed change
1. **API**: add optional artefact filters to `IssueTestResultAttachmentRulePostRequest` and the rule-matching logic — at minimum `artefacts` (names); ideally also `artefact_tracks`, `artefact_versions`, `artefact_stages` to mirror the dimensions already returned by `GET .../attachment-rule-filters` under `artefact_filters`.
2. **MCP server (`canonical/test-observer-mcp`)**: expose the new filter(s) in the `create_attachment_rule` tool (add `artefacts` to `createAttachmentRuleInput`) and forward them.

## Example enabled after the change
```
create_attachment_rule(
issue_id=,
artefacts=["self-signed-certificates"],
test_case_names=["test_build_bundle"],
test_result_statuses=["FAILED"],
)
```
Matches every SSC bundle-build failure regardless of which neighbor endpoint is unfulfilled, without touching any other charm.

## Acceptance criteria
- A rule with `artefacts=[X]` matches only results whose artefact name is X.
- Combinable with existing filters (test_case, families, status, execution_metadata).
- `matching_test_results_count` reflects the artefact scope.
- MCP `create_attachment_rule` accepts and forwards `artefacts` (+ optional track/version/stage).

## Caveat
Artefact-scoped rules should still generally be paired with a failure-signature dimension (test_case / template / execution_metadata) so a rule doesn't attach an issue to *all* of a charm's failures across unrelated bugs.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the IssueTestResultAttachmentRulePostRequest schema and the attachment-rule matching logic, then inspect the artefact_filters returned by the attachment-rule-filters endpoint. Update the MCP createAttachmentRuleInput and create_attachment_rule forwarding path as needed. Done means artefact filters combine with existing filters, matching_test_results_count respects the scope, and only results for the selected artefact are matched.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.