BUG: ScorerMetrics.to_json() raises TypeError on the trial_scores array ScorerEvaluator attaches
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
#### Describe the bug
`ScorerMetrics.to_json()` raises `TypeError` on exactly the metrics objects that
`ScorerEvaluator` returns, because `trial_scores` is a numpy array and `json.dumps` cannot
encode one.
`to_json()` documents itself as "the canonical serialization entry point for ``ScorerMetrics``
and its subclasses", paired with `from_json_file()` "for round-trip (de)serialization"
(`pyrit/score/scorer_evaluation/scorer_metrics.py:51-63`). The evaluator deliberately attaches
that array to the object it hands back
(`pyrit/score/scorer_evaluation/scorer_evaluator.py:448-450`):
```python
# Include trial scores for debugging and future mismatch analysis
# (not persisted to registry - use returned metrics object for detailed analysis)
metrics.trial_scores = all_model_scores
```
so the object the in-source comment points callers at for "detailed analysis" is the one object
the documented serializer refuses to serialize. `from_json_file()` filters out only
underscore-prefixed keys, i.e. it is written as though `trial_scores` were part of what
`to_json()` emits.
Two things this is *not*, so the scope is clear:
- The JSONL registry path is unaffected. `_metrics_to_registry_dict()`
(`pyrit/score/scorer_evaluation/scorer_metrics_io.py:42-58`) excludes `trial_scores`, and
`tests/unit/score/test_scorer_metrics_io.py:136` pins that exclusion. `evaluate_async()` and
registry reads work today.
- No in-tree code and no documentation example hits it. The only callers of `metrics.to_json()`
in the repo are the two round-trip tests in `tests/unit/score/test_scorer_metrics.py`
(`:36` and `:57`), and both construct metrics without `trial_scores` — which is why the suite
is green while the pair's documented contract is broken for real evaluator output.
#### Steps/Code to Reproduce
```python
import numpy as np
from pyrit.score import ObjectiveScorerMetrics
metrics = ObjectiveScorerMetrics(
num_responses=2,
num_human_raters=1,
accuracy=0.9,
accuracy_standard_error=0.05,
f1_score=0.8,
precision=0.85,
recall=0.75,
trial_scores=np.array([[0.2, 0.4], [0.2, 0.4]]),
)
print(metrics.to_json())
```
#### Expected Results
A JSON string, `"trial_scores": [[0.2, 0.4], [0.2, 0.4]]`, that `from_json_file()` reads back
into the same shape — as it already does for every other field.
#### Actual Results
```
File ".../python3.14/json/encoder.py", line 182, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
f'is not JSON serializable')
TypeError: Object of type ndarray is not JSON serializable
when serializing dict item 'trial_scores'
```
Same for `HarmScorerMetrics`, and reached without hand-building anything: running the real
`HarmScorerEvaluator.evaluate_dataset_async(...)` with two trials (the mocked-scorer recipe from
`tests/unit/score/test_scorer_evaluator.py:75`) returns metrics whose fields are
```
trial_scores: type=ndarray dtype=float64 value=array([[0.2, 0.4],
[0.2, 0.4]])
mean_absolute_error: type=float64 dtype=float64 value=np.float64(0.0)
```
and then `metrics.to_json()` raises the identical `TypeError`. Note that the `np.float64` fields
are *not* the problem: `np.float64` subclasses `float`, so `json.dumps` handles them; `ndarray`,
`np.int64` and `np.bool_` are the numpy types it rejects (measured on numpy 2.4.4).
#### Versions
- OS: macOS 27.0.0 (arm64)
- Python version: 3.14.5
- PyRIT version: 1.2.0.dev0, from `main` at `543c20c`
- numpy 2.4.4
#### Proposed fix
If maintainers agree this is worth fixing, I'd send a small PR: give `to_json()` a `default=`
hook that encodes numpy arrays/scalars with `.tolist()`, and have `from_json_file()` restore
`trial_scores` as an `np.ndarray` so the round trip returns the declared type instead of a nested
list. Tests: one regression test per metrics subclass with `trial_scores` populated, plus one
asserting the hook still raises for values that are neither JSON nor numpy (so the fix does not
turn the crash into silent stringification).
One adjacent thing I am deliberately **not** folding in, but flagging: `==` between two
array-carrying metrics raises
`ValueError: The truth value of an array with more than one element is ambiguous`, because the
dataclass compares `trial_scores` elementwise. That means `assert loaded == metrics` — the oracle
the existing two round-trip tests use — cannot be applied to an object with trial scores at all,
and a fix would be a semantics decision (`field(compare=False)` vs. something else) rather than a
bug fix. Happy to open a separate issue for it if useful.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pyrit/score/scorer_evaluation/scorer_metrics.py at to_json() and from_json_file(), then review how scorer_evaluator.py attaches trial_scores. Use the reproduction and add regression coverage alongside tests/unit/score/test_scorer_metrics.py for both metrics subclasses. Done means evaluator-produced metrics serialize to JSON and restore trial_scores with its expected array shape, while unsupported values still fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100