a2ui-project / a2ui-project/a2ui

[FEATURE]: Classify a2ui_scorer failures instead of collapsing them to a single 0.0

Ouverte
#2,200 1 commentaire 0 réactions 1 personne assignée Réclamée par @Varun-S10 Voir sur GitHub
P2 status: first-line-handled type: feature/enhancement
Langage dominant
TypeScript
Étoiles
16.4k
Forks
1.3k
Merge moyen
3 j 15 h
PR mergées (30 j)
134

Description

- [x] I have searched the existing issues to make sure this feature has not already been requested.

## Is your feature request related to a problem? Please describe.

`a2ui_scorer` in `eval/a2ui_eval/scorers.py` reduces every failure to the same thing:

```python
except Exception as e:
return Score(value=0.0, answer=answer_text, explanation=str(e))
```

With `metrics=[accuracy()]`, the only thing that reaches a run summary is a pass rate. The per-sample `explanation` is in the log, but nothing aggregates it, so there is no way to ask "what kind of failures did this run produce?"

The committed baseline shows why that hurts. `eval/baselines/direct_json/run_meta.json` covers 51 samples, 5 of which fail:

| sample | schema_acc | quality_acc | code_tokens |
| --- | --- | --- | --- |
| productGalleryData | 1.0 | 0.0 | 802 |
| weatherForecast | 1.0 | 0.0 | 992 |
| standardFunctions | 1.0 | 0.0 | 446 |
| restaurantFinder | 0.0 | 0.0 | 1705 |
| rizzCharts | 0.0 | 0.0 | 16 |

`restaurantFinder` emitted 1705 code tokens and failed schema validation. `rizzCharts` emitted 16 and failed. Those are opposite problems. One is a rich payload that violates the schema, the other is essentially no answer at all. Both land as `schema_acc: 0.0`.

This matters more now that the iterative format optimizer (#2092) compares `direct`, `express`, `elemental` and `atom` against exactly these aggregates. When `schema_acc` moves between formats, there is no way to tell whether the format made models emit more malformed payloads or more empty responses. Those call for opposite fixes.

## Describe the Proposed Solution

Stop discarding the classification the SDK already produces.

`a2ui.core.exceptions` already defines a typed hierarchy: `A2uiParseError`, `A2uiValidationError`, `A2uiIntegrityError`, `A2uiRecursionError`, `A2uiCatalogError` and `A2uiCompileError`. On top of that, `A2uiValidationError` carries `A2uiErrorDetail(path, code, message)` with codes including `missing_field`, `type_mismatch`, `extra_field`, `invalid_value` and `invalid_pointer`. `validation/validator.py` builds all of it, and the scorer flattens the lot to `str(e)`.

Proposed:

1. Derive a coarse category from the exception type, and a finer one from `detail.code`.
2. Give the three branches that already exist before validation their own categories: no model output, solver-step compilation failure, and no A2UI payload found in the response. That last one is very likely what `rizzCharts` is.
3. Record the category on `Score.metadata` and add a small `@metric` that reports the distribution over a run.

Deliberately additive. `Score.value` stays 0.0/1.0 and `accuracy()` is untouched, so `schema_acc` keeps its current meaning and existing baselines stay comparable.

## Describe Alternatives Considered

- **Dict-valued score with per-key metrics.** Changes the shape of the metric output, which would break comparison against the committed baselines and `bin/report_evals.py`.
- **A second scorer emitting a categorical value with `frequency()`.** `frequency()` is built for exactly this, but a separate scorer would re-run parsing and validation only to label a failure that has already been computed once.
- **Leave it, and parse `explanation` strings downstream.** Fragile: those strings come from jsonschema and change whenever the schema does.

## Additional Context

No new dependencies are needed; everything required is already in the SDK and in Inspect. The pass/fail value does not change, so this should be safe to land ahead of further v1.0 eval work.

Happy to put up a PR with tests if you think this is worth having. I would want to agree the category names with you first, since they would end up in run metadata that other tooling might key off.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez dans eval/a2ui_eval/scorers.py, au niveau de la fonction a2ui_scorer. Examinez le bloc de gestion des exceptions ainsi que la hiérarchie d’exceptions existante dans a2ui.core.exceptions. L’objectif est de classer les échecs par catégories en fonction du type d’exception et de detail.code, puis d’ajouter cette classification à Score.metadata. Écrivez une petite métrique qui indique la distribution. Testez en exécutant les scripts d’évaluation existants afin de vous assurer que les valeurs de pass/fail restent inchangées.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
backend-api-design, testing-qa
Type d'issue
Fonctionnalité
Difficulté
3/5
Temps estimé
1-2 jours
Activité
Calme
Clarté
Clairement spécifiée
Accessibilité débutants
65/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.