microsoft / microsoft/PyRIT

ConversationScorer includes raw tool output in the scored conversation text although its comments say it is excluded

Open
#2,706 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.5k
Forks
893
Avg merge
3d 50m
Merged PRs (30d)
165

Description

#### Describe the bug

`ConversationScorer` builds the text that a harm/objective scorer judges, and three consecutive
comments in that function say raw tool output is deliberately left out of it — but the code includes
it, so a `role="tool"` piece is folded into the scored conversation and rendered as `"Tool: …"`.

`pyrit/score/conversation_scorer.py` on `main`:

```python
# Goes through each message in the conversation and appends user/assistant messages only
# Explicitly excludes system, tool, developer messages from being scored/included in conversation history
# they are allowed in validation but not included in the scored conversation text
for conv_message in conversation:
for piece in conv_message.message_pieces:
# Only include user and assistant messages in the conversation text
if piece.api_role in ["user", "assistant", "tool"] and self._validator.is_role_supported(piece):
```

The validator does not close the gap: `ScorerPromptValidator`'s `supported_roles` "Defaults to every
role except `simulated_assistant`" (`pyrit/score/scorer_prompt_validator.py:45-47`), so under the
default validator a tool piece passes both conditions.

#### Steps/Code to Reproduce

Offline unit-style reproduction (no keys, no network), following
`tests/unit/score/test_conversation_history_scorer.py::test_conversation_history_scorer_filters_roles_correctly`
and printing what the wrapped scorer actually receives. Pieces stored for one conversation id:
`user`, `tool`, `developer`, `system`, `assistant`, each with a distinct sentinel string; then
capture `mock_scorer._score_nested_async.call_args.kwargs["scorable"].value`.

Measured on `microsoft/PyRIT` `main` @ `2215c5b` (Python 3.14.5, `uv sync` default groups):

```
SCORED TEXT repr:
'User: USER_SENTINEL\nTool: TOOL_SENTINEL\nAssistant: ASSISTANT_SENTINEL\n'
USER_SENTINEL in scored text -> True
TOOL_SENTINEL in scored text -> True
DEV_SENTINEL in scored text -> False
SYS_SENTINEL in scored text -> False
ASSISTANT_SENTINEL in scored text -> True
```

The neighbouring test already pins the intended shape for two of the roles — it asserts
`expected_conversation = "User: User message\nAssistant: Assistant message\n"` and
`assert "System message" not in called_scorable.value` — and it does not cover a tool piece at all,
which is why the divergence is invisible to CI.

#### Expected Results

One of the two, and this is the question I cannot answer from the code alone:

- either tool output is excluded, matching the three comments (drop `"tool"` from the role list), or
- tool output is included on purpose, in which case the comments are wrong and a test should pin the
behaviour so the two cannot drift again.

#### Actual Results

Tool text is included in the conversation the scorer judges, while `developer` and `system` are not.

#### Why I think it matters for scoring correctness

A scorer is being asked to judge what the *target* produced. Third-party tool results are neither a
user turn nor an assistant turn, so a long or hostile tool payload sitting in the judged text can move
the verdict in either direction without reflecting the target's own behaviour. `scorer_prompt_validator.py:9-14`
gives the analogous reason for dropping simulated history — a scorer that judges the target must not
mistake it for what the target said.

The history explains the split rather than settling it: the comments came in with the original
`ConversationScorer` (#1138, `2337fce7`, 2025-12-12), while `"tool"` was added to the whitelist by
#2518 "Policy Scorer Compatibility (phase 2.5)" (`8b3826be`, 2026-09-02), which left the adjacent
comments untouched. So I cannot tell whether #2518 intended policy scorers to see tool output and the
comments simply went stale, or whether the widening was incidental.

If tool visibility is required only for the policy-scorer path, a validator-driven option is to keep
the exclusion in `ConversationScorer` and let a scorer that wants tool output declare it through
`supported_roles`, so the include/exclude decision lives in one place instead of two.

Not verified, and I would not want a fix to assume it: I have not checked whether any shipped scenario
actually scores a conversation that contains tool pieces, so I cannot say how often this changes a real
verdict in practice — only that the default path includes them.

#### Versions

- OS: macOS 27.0 (arm64)
- Python: 3.14.5 (also reproduced logic review on 3.11.15)
- PyRIT: 1.2.0.dev0 from `main` @ `2215c5b`

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read pyrit/score/conversation_scorer.py and pyrit/score/scorer_prompt_validator.py, then run the neighboring test in tests/unit/score/test_conversation_history_scorer.py with a tool-role piece added. Check with maintainers whether tool output should be scored, and align the implementation, comments, and regression coverage with that decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.