reporting: record detector-scoped output representation in hitlog for translated runs

Summary

On translated runs, a language-specific detector can score a reverse-translated output while the corresponding hitlog entry records only the raw model output.

The generation index remains aligned (including after #1959), so this is not a wrong-generation attribution bug. The auditability problem is that the hitlog does not contain the output representation selected for detection. A hit can therefore be difficult to explain or reproduce from the hitlog alone.

Current behavior

Language-aware detectors commonly select inputs through Attempt.outputs_for(self.lang_spec). For example, StringDetector.detect() uses:

all_outputs = attempt.outputs_for(self.lang_spec)

When the requested detector language differs from the prompt/model language, outputs_for() returns reverse_translation_outputs.

The evaluator later enumerates the resulting scores but writes:

"output": asdict(attempt.outputs[idx]),

Consequently:

• score may have been calculated from reverse_translation_outputs[idx];
• output contains attempt.outputs[idx], the raw model generation;
• the hitlog does not identify or include the representation used for detection.

Relevant code:

• https://github.com/NVIDIA/garak/blob/main/garak/detectors/base.py#L221-L249
• https://github.com/NVIDIA/garak/blob/main/garak/attempt.py#L397-L411
• https://github.com/NVIDIA/garak/blob/main/garak/evaluators/base.py#L77-L128

Basic example

Construct an attempt whose model-facing language differs from an English-only detector:

attempt = Attempt(prompt=Message("prompt", lang="es"))
attempt.outputs = [Message("respuesta sin la palabra inglesa", lang="es")]
attempt.reverse_translation_outputs = [
    Message("reverse translation containing TRIGGER", lang="en")
]

detector = StringDetector(["TRIGGER"])
detector.lang_spec = "en"
scores = detector.detect(attempt)

The detector scores the English reverse translation and produces a hit. After assigning the score and evaluating the attempt, the hitlog's output field contains the Spanish model output. It does not contain the English representation where TRIGGER matched, nor metadata identifying the representation used for detection.

The full report serializes reverse_translation_outputs, so the evidence can sometimes be reconstructed by joining records using the attempt ID and index. The standalone hitlog record is not self-explanatory.

Motivation

Keep output as the actual model generation, but make a translated-run hit independently auditable by recording the detector-scoped output selected before detector-specific preprocessing.

One possible backward-compatible shape would be an optional field such as:

{
  "output": {"text": "raw model generation", "lang": "es"},
  "detector_input": {"text": "reverse-translated representation", "lang": "en"},
  "detector_input_source": "reverse_translation_outputs"
}

The field names and storage location are open for discussion.

Design consideration

The evaluator currently receives only detector_name and scores; it does not receive the detector instance or its lang_spec. Correctly identifying the selected representation may therefore require carrying detector-input provenance through the harness/attempt data rather than having the evaluator infer it after the fact.

I would appreciate maintainer guidance on the preferred representation before preparing a patch.

Dedupe notes

• #1959 fixed the former ordering bug and establishes position-for-position alignment; this proposal does not change that behavior.
• #1174 tracks broader multilingual cleanup but does not appear to cover hitlog evidence/provenance.
• #1201 and #1826 concern whether reverse-translation fields are serialized when translation is unused, rather than which representation a detector score was based on.

Read garak/detectors/base.py, garak/attempt.py, and garak/evaluators/base.py, starting with outputs_for(), detector scoring, and hitlog serialization. Trace how detector-input selection and score indexing flow through the evaluator; done means a translated-run hitlog can identify the detector-scoped representation used for scoring while retaining the raw model output.