RFC: Structured Evaluator Uncertainty and Error Semantics

1. Context & Motivation

Haystack already provides a mature evaluation layer for RAG and LLM application workflows.

The current evaluator stack includes components such as:

• LLMEvaluator;
• ContextRelevanceEvaluator;
• FaithfulnessEvaluator;
• EvaluationRunResult.

These components already expose useful evaluation information through score, individual_scores, and evaluator-specific results.

However, some evaluator states remain compressed into scalar values, None, empty placeholders, or nan.

This RFC proposes a narrow evaluator/reporting improvement: make unresolved evaluator states explicit without changing Haystack's existing score-first evaluation model.

The goal is not to add a broad theory layer to Haystack. The goal is to help users distinguish between:

• a valid supported result;
• a valid unsupported result;
• an unresolved evaluator judgment;
• an evaluator execution or parsing error.

2. Problem Statement

In evaluation workflows, not all non-positive or missing results mean the same thing.

A row may fail because:

1. the evaluated answer is unsupported;
2. the evaluated context is irrelevant;
3. the LLM evaluator returned malformed JSON;
4. the evaluator output did not contain the expected fields;
5. the evaluator call failed;
6. the row could not be judged confidently.

Today, these states can become difficult to separate once they are represented as:

• None;
• empty result placeholders;
• float("nan");
• aggregate score;
• individual_scores.

This can weaken evaluation reports.

For production users, a low score and an invalid evaluator row require different actions. A low score may suggest retrieval or generation quality issues. An invalid evaluator row may suggest prompt, parser, model, or infrastructure issues.

The current reporting shape can make that distinction harder than necessary.

3. Proposed Solution

Introduce an optional structured evaluator-status channel for LLM evaluator outputs and evaluation reporting.

This should preserve the existing contract:

• keep score;
• keep individual_scores;
• keep evaluator-specific results;
• keep existing scalar-first workflows working.

The additional information should make unresolved states explicit.

Conceptual example:

{
    "score": 0.75,
    "individual_scores": [1.0, 0.0, float("nan")],
    "results": [...],
    "statuses": ["supported", "unsupported", "indeterminate"],
    "error_count": 1,
    "indeterminate_count": 1,
}

The exact field names should be decided by Haystack maintainers.

The key design principle is additive reporting: no breaking change, no global evaluator redesign, no mandatory new abstraction.

4. Academic Foundation

This proposal is inspired by neutrosophic logic, introduced by Prof. Florentin Smarandache.

Neutrosophic logic separates evaluation into three independent dimensions:

• Truth: evidence supporting a proposition;
• Indeterminacy: incomplete, ambiguous, unresolved, or non-decidable evidence;
• Falsity: evidence against a proposition.

For Haystack, this RFC translates the theory into practical evaluator language:

• supported or relevant corresponds to evidence for the output;
• unsupported or irrelevant corresponds to evidence against the output;
• indeterminate corresponds to unresolved evaluator judgment;
• error corresponds to evaluator execution, parsing, or schema failure.

The public API does not need to use the word neutrosophic. The concept can remain academically grounded while the implementation uses Haystack-native terminology such as evaluator status, uncertainty, or error semantics.

Relevant references from the same neutrosophic research lane used in prior projects:

1. Florentin Smarandache
   Neutrosophy: Neutrosophic Probability, Set, and Logic
   Foundation of neutrosophic logic, set theory, probability, and statistics.

2. Florentin Smarandache & Maissam Ahmad Jdid
   Neutrosophic Linear Models and Algorithms to Find Their Optimal Solution
   Relevant for optimization and decision-making under incomplete information.

3. Maikel Yelandi Leyva-Vázquez & Florentin Smarandache
   Breaking the Chains of Probability: Neutrosophic Logic as a New Framework for Epistemic Uncertainty in Large Language Models
   Relevant to LLM evaluation because it studies truth, indeterminacy, and falsity in LLM outputs.

4. Smarandache, Ye, and collaborators
   Work on neutrosophic multi-criteria decision-making and aggregation operators.

5. mleyvaz/neutrosophic-llm-logic
   Practical reference implementation for prompting LLMs to produce truth/indeterminacy/falsity evaluations.
   DOI: 10.5281/zenodo.19911845

5. Why Haystack

Haystack is a good fit because the relevant extension points already exist.

This RFC does not require introducing a new framework-wide system. It can attach to evaluator and reporting surfaces that already exist:

• LLMEvaluator already handles structured evaluator outputs;
• ContextRelevanceEvaluator already reduces LLM evaluator judgments into score-oriented outputs;
• FaithfulnessEvaluator already works with statement-level support judgments;
• EvaluationRunResult already provides aggregate and detailed reports.

This makes the evaluator layer the safest first place to discuss uncertainty/status semantics.

Other possible areas, such as retriever confidence, HITL decision states, agent guardrails, or audit receipts, may be interesting later. They are not the right first target because they would broaden the proposal before the evaluator/reporting value is validated.

6. Proposed Plan

Phase 0 - RFC issue first

Open this as a maintainer-facing RFC issue before implementation.

The issue should ask whether maintainers agree that evaluator failures, unresolved evaluator rows, and valid negative judgments should be reported separately.

No code should be pushed upstream before this discussion exists.

The RFC issue should explicitly ask maintainers to guide:

• naming;
• result shape;
• whether EvaluationRunResult is the right reporting surface;
• whether this should attach to LLMEvaluator, concrete evaluators, or both;
• whether the implementation should avoid all public neutrosophic naming.

Phase 1 - Narrow implementation branch and PR

If the direction is acceptable, create a Phase 1 branch.

Suggested branch name:

feature/haystack-evaluator-uncertainty-phase1

Phase 1 should be the only upstream PR candidate at first.

It should focus on the smallest reviewable unit:

• preserve score;
• preserve individual_scores;
• add explicit evaluator row status or error visibility;
• add focused tests;
• avoid retriever, agent, HITL, router, or governance changes.

The Phase 1 PR should link back to the RFC issue and clearly state that it is not trying to redesign Haystack evaluation.

After the Phase 1 PR is pushed, protect the branch on the fork:

• disable force-push;
• disable deletion;
• preserve linear history if available;
• keep review comments and branch state stable.

Phase 2 - Local/fork branch only until maintainer feedback

Prepare Phase 2 as a stacked local or fork branch, but do not treat it as final.

Suggested branch name:

feature/haystack-evaluator-reporting-phase2

Phase 2 should depend on what maintainers say about Phase 1.

Likely Phase 2 direction:

• improve EvaluationRunResult reporting;
• expose aggregate counts for valid, indeterminate, and error rows;
• improve detailed reports without changing score semantics.

Phase 2 should not be opened as a normal upstream PR until one of these happens:

• Phase 1 receives maintainer feedback asking for reporting work;
• Phase 1 is accepted or merged;
• maintainers explicitly ask to see the stacked branch.

The branch may be pushed to the fork and protected as evidence of feasibility, but it should be clearly marked as dependent on Phase 1 review.

Phase 3 - Local/fork exploratory branch only

Prepare Phase 3 as a local or fork-visible proof branch, not as an upstream PR.

Suggested branch name:

feature/haystack-statement-uncertainty-phase3

Likely Phase 3 direction:

• explore statement-level uncertainty for evaluators such as FaithfulnessEvaluator;
• distinguish supported, unsupported, indeterminate, and error states at a finer granularity;
• test whether this improves diagnostic value.

Phase 3 is expected to change after maintainer feedback.

It should not be presented as a fixed roadmap. It should be treated as a research branch that proves feasibility and helps us respond intelligently if maintainers ask for deeper evaluator semantics.

Protect the branch if pushed to the fork, but do not pressure maintainers with it.

Phase 4 - Optional future discussion only

Any retriever confidence, router fallback, HITL, agent, or governance work should remain outside this RFC.

If maintainers like the evaluator-status model, a later RFC can discuss whether similar concepts belong elsewhere.

That future work should not be bundled with this proposal.

7. Explicit Constraints

This RFC does not propose:

• replacing existing evaluator scores;
• removing individual_scores;
• breaking current evaluator outputs;
• changing pipeline execution;
• changing retriever score semantics;
• adding a mandatory uncertainty framework;
• adding signed receipts;
• adding a governance middleware;
• changing agent or HITL behavior;
• forcing Haystack to expose neutrosophic terminology in public APIs.

The first public contribution should remain evaluator/reporting-only.

8. Contribution Workflow

The workflow should follow the disciplined Ragas-style case-study pattern:

1. Open the RFC issue first.
2. Wait for initial maintainer signal or at least make the proposal publicly inspectable.
3. Create Phase 1 branch from the correct base.
4. Implement only the narrow Phase 1 evaluator-status change.
5. Add focused tests.
6. Push Phase 1 branch to the fork.
7. Open one Phase 1 PR linked to the RFC issue.
8. Protect the Phase 1 branch.
9. Prepare Phase 2 as a stacked local/fork branch only.
10. Protect Phase 2 if pushed to fork.
11. Prepare Phase 3 as a local/fork proof branch only.
12. Protect Phase 3 if pushed to fork.
13. Do not open Phase 2 or Phase 3 upstream until maintainer feedback changes or confirms the path.
14. Update the RFC issue with concise milestone comments only when there is real progress.
15. Stop when the next honest action depends on maintainer response.

This workflow keeps implementation momentum without creating pressure on maintainers to review a large stack of speculative changes.

9. Prior Art & Uniqueness

Prior art exists in:

• scalar RAG evaluation metrics;
• LLM-as-judge workflows;
• evaluator error handling;
• faithfulness and relevance scoring;
• uncertainty-aware decision-making;
• neutrosophic LLM evaluation research.

The unique contribution here is not simply adding another score.

The unique contribution is making evaluator uncertainty operational inside Haystack's existing evaluation surfaces while preserving compatibility with existing score-based users.

Compared with prior neutrosophic study cases:

• Agent Squad applied the concept to multi-agent response fusion.
• Aden/Hive applied it to worker/Queen swarm decision visibility.
• Ragas applied it to RAG evidence-triplet evaluation.
• Haystack would apply it to evaluator status and evaluation reporting.

That makes the proposal narrower and more natural for Haystack.

10. Open Questions For Maintainers

1. Is explicit evaluator row status useful for Haystack users?
2. Should unresolved evaluator states be reported separately from valid negative judgments?
3. Should this live in evaluator outputs, meta, EvaluationRunResult, or a combination?
4. What names best fit Haystack conventions: statuses, row_statuses, evaluation_statuses, errors, or something else?
5. Should the first PR target only LLMEvaluator, or also concrete evaluators like ContextRelevanceEvaluator and FaithfulnessEvaluator?
6. Should failed rows remain represented as nan while status fields explain why?
7. Should public API naming avoid the term neutrosophic entirely?
8. Would maintainers prefer this as a small code contribution, a recipe/example, or only a documentation discussion first?

11. Next Step

The next step is to open the RFC issue and ask for maintainer direction.

If maintainers are open to the idea, the first implementation should be a narrow Phase 1 PR that:

• keeps existing score semantics intact;
• adds explicit evaluator row status/error visibility;
• includes focused tests;
• links back to the RFC issue;
• avoids all retriever, agent, router, HITL, and governance changes.

Phase 2 and Phase 3 should be prepared only as protected local/fork branches until maintainer feedback determines whether they should be changed, replaced, or discarded.

Start by reading the existing LLMEvaluator, ContextRelevanceEvaluator, FaithfulnessEvaluator, and EvaluationRunResult surfaces to understand how score, individual_scores, and results are reported. Use the maintainer discussion to resolve naming, result shape, and ownership before implementation. Done for this RFC means agreement on a narrow, additive Phase 1 change with focused tests and no broader evaluator redesign.