agentscope-ai / agentscope-ai/OpenJudge

[Bug]:Analyzers silently ignore failed or missing grader results and may report misleading metrics

オープン
#195 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug
主要言語
Python
スター
830
フォーク
69
平均マージ
5日 8時間
マージ済み PR(30日)
4

説明

## Problem

Several analyzers silently exclude failed or missing grader results when calculating evaluation metrics. This can produce overly optimistic results without warning users that only a subset of the dataset was analyzed.

For example, given three samples where only one grading request succeeds:

```python
from openjudge.analyzer.statistical import DistributionAnalyzer
from openjudge.analyzer.validation import AccuracyAnalyzer
from openjudge.graders.schema import GraderError, GraderScore

dataset = [
{"label": 1},
{"label": 0},
{"label": 0},
]

results = [
GraderScore(name="judge", score=1, reason="ok"),
GraderError(name="judge", reason="timeout", error="timeout"),
GraderError(name="judge", reason="rate limited", error="429"),
]

distribution = DistributionAnalyzer().analyze(dataset, results)
accuracy = AccuracyAnalyzer().analyze(dataset, results)

print(distribution.mean)
print(accuracy.accuracy)
```

The current output is effectively:

```text
Mean score: 1.0
Accuracy: 1.0
```

Both failed evaluations are skipped, so a run with only 33% valid coverage appears to have a perfect score and perfect accuracy.

There is a related data-alignment problem. Validation analyzers pair the dataset and results with `zip()` without checking their lengths:

```python
AccuracyAnalyzer().analyze(dataset, results[:1])
```

This also reports 100% accuracy based on one result, while silently ignoring the other two dataset samples.

This behavior can introduce survivorship bias into evaluation reports. If difficult samples are more likely to time out, be rate-limited, or fail parsing, excluding them can make an unreliable grader or model appear substantially better than it is.

## Affected components

The issue is visible in at least:

- `DistributionAnalyzer`, which skips results without a score.
- `AccuracyAnalyzer`, which skips non-`GraderScore` results.
- Other validation analyzers that use the same `zip(dataset, grader_results)` pattern, including correlation, precision, recall, and F1 analysis.
- `PairwiseAnalyzer`, which skips invalid comparisons when calculating wins but reports `total_comparisons=len(grader_results)`, including results that were not actually analyzed.

## Expected behavior

Analyzers should make incomplete evaluation coverage explicit and should not silently accept misaligned datasets and result lists.

One possible solution is:

1. Validate that `len(dataset) == len(grader_results)` for analyzers that require one result per sample. Raise a clear `ValueError` when the inputs are misaligned.
2. Include coverage information in analysis metadata or result schemas:
- `total_samples`
- `valid_results`
- `error_results`
- `skipped_results`
- `coverage_rate`
3. Ensure that explanations clearly state when metrics are calculated from only a subset of the dataset.
4. Report the number of valid pairwise comparisons separately from the number of submitted results.

An optional `error_policy` could make the behavior configurable:

- `"exclude"`: exclude failed results from the metric, but report coverage prominently.
- `"count_as_incorrect"`: count failed results as incorrect where applicable.
- `"raise"`: stop analysis when any failed result is present.

For backward compatibility, `"exclude"` could remain the default as long as the reduced coverage is clearly exposed. Length mismatches should still fail early because positional alignment cannot be guaranteed.

## Additional context

This issue affects evaluation correctness rather than only presentation. The current APIs return valid-looking metric objects, so downstream automation may consume inflated scores without noticing that most samples failed or were omitted.

If the maintainers agree with the proposed direction, I would be happy to submit a pull request with the validation, coverage metadata, and regression tests.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。