microsoft / microsoft/PyRIT

BUG: ScorerMetrics.to_json() raises TypeError on the trial_scores array ScorerEvaluator attaches

Open
#2,710 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.5k
Forks
893
Avg merge
3d 50m
Merged PRs (30d)
165

Description

#### Describe the bug

`ScorerMetrics.to_json()` raises `TypeError` on exactly the metrics objects that
`ScorerEvaluator` returns, because `trial_scores` is a numpy array and `json.dumps` cannot
encode one.

`to_json()` documents itself as "the canonical serialization entry point for ``ScorerMetrics``
and its subclasses", paired with `from_json_file()` "for round-trip (de)serialization"
(`pyrit/score/scorer_evaluation/scorer_metrics.py:51-63`). The evaluator deliberately attaches
that array to the object it hands back
(`pyrit/score/scorer_evaluation/scorer_evaluator.py:448-450`):

```python
# Include trial scores for debugging and future mismatch analysis
# (not persisted to registry - use returned metrics object for detailed analysis)
metrics.trial_scores = all_model_scores
```

so the object the in-source comment points callers at for "detailed analysis" is the one object
the documented serializer refuses to serialize. `from_json_file()` filters out only
underscore-prefixed keys, i.e. it is written as though `trial_scores` were part of what
`to_json()` emits.

Two things this is *not*, so the scope is clear:

- The JSONL registry path is unaffected. `_metrics_to_registry_dict()`
(`pyrit/score/scorer_evaluation/scorer_metrics_io.py:42-58`) excludes `trial_scores`, and
`tests/unit/score/test_scorer_metrics_io.py:136` pins that exclusion. `evaluate_async()` and
registry reads work today.
- No in-tree code and no documentation example hits it. The only callers of `metrics.to_json()`
in the repo are the two round-trip tests in `tests/unit/score/test_scorer_metrics.py`
(`:36` and `:57`), and both construct metrics without `trial_scores` — which is why the suite
is green while the pair's documented contract is broken for real evaluator output.

#### Steps/Code to Reproduce

```python
import numpy as np
from pyrit.score import ObjectiveScorerMetrics

metrics = ObjectiveScorerMetrics(
num_responses=2,
num_human_raters=1,
accuracy=0.9,
accuracy_standard_error=0.05,
f1_score=0.8,
precision=0.85,
recall=0.75,
trial_scores=np.array([[0.2, 0.4], [0.2, 0.4]]),
)
print(metrics.to_json())
```

#### Expected Results

A JSON string, `"trial_scores": [[0.2, 0.4], [0.2, 0.4]]`, that `from_json_file()` reads back
into the same shape — as it already does for every other field.

#### Actual Results

```
File ".../python3.14/json/encoder.py", line 182, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
f'is not JSON serializable')
TypeError: Object of type ndarray is not JSON serializable
when serializing dict item 'trial_scores'
```

Same for `HarmScorerMetrics`, and reached without hand-building anything: running the real
`HarmScorerEvaluator.evaluate_dataset_async(...)` with two trials (the mocked-scorer recipe from
`tests/unit/score/test_scorer_evaluator.py:75`) returns metrics whose fields are

```
trial_scores: type=ndarray dtype=float64 value=array([[0.2, 0.4],
[0.2, 0.4]])
mean_absolute_error: type=float64 dtype=float64 value=np.float64(0.0)
```

and then `metrics.to_json()` raises the identical `TypeError`. Note that the `np.float64` fields
are *not* the problem: `np.float64` subclasses `float`, so `json.dumps` handles them; `ndarray`,
`np.int64` and `np.bool_` are the numpy types it rejects (measured on numpy 2.4.4).

#### Versions

- OS: macOS 27.0.0 (arm64)
- Python version: 3.14.5
- PyRIT version: 1.2.0.dev0, from `main` at `543c20c`
- numpy 2.4.4

#### Proposed fix

If maintainers agree this is worth fixing, I'd send a small PR: give `to_json()` a `default=`
hook that encodes numpy arrays/scalars with `.tolist()`, and have `from_json_file()` restore
`trial_scores` as an `np.ndarray` so the round trip returns the declared type instead of a nested
list. Tests: one regression test per metrics subclass with `trial_scores` populated, plus one
asserting the hook still raises for values that are neither JSON nor numpy (so the fix does not
turn the crash into silent stringification).

One adjacent thing I am deliberately **not** folding in, but flagging: `==` between two
array-carrying metrics raises
`ValueError: The truth value of an array with more than one element is ambiguous`, because the
dataclass compares `trial_scores` elementwise. That means `assert loaded == metrics` — the oracle
the existing two round-trip tests use — cannot be applied to an object with trial scores at all,
and a fix would be a semantics decision (`field(compare=False)` vs. something else) rather than a
bug fix. Happy to open a separate issue for it if useful.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pyrit/score/scorer_evaluation/scorer_metrics.py at to_json() and from_json_file(), then review how scorer_evaluator.py attaches trial_scores. Use the reproduction and add regression coverage alongside tests/unit/score/test_scorer_metrics.py for both metrics subclasses. Done means evaluator-produced metrics serialize to JSON and restore trial_scores with its expected array shape, while unsupported values still fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.