[Discussion] Project Reliance — an idea for adaptive AI verification

Hi PyTorch team,

I’m working on a research idea called Project Reliance.

I’m interested in a simple question:

What if an AI did not always use the strongest verification system, but could increase the amount of checking when an answer is difficult, uncertain, or risky?

I’m currently working on the idea and mathematical design. I have not implemented the full system yet, so I’m sharing the idea early and would like feedback before going further.

The basic idea

The rough architecture is:

Input
↓
AI creates a candidate answer
↓
Evidence Module (EM)
↓
Evidence Reasoner (ER)
↓
Is the evidence enough?
├── Yes → Accept
└── No → stronger verification
↓
AIC
↓
Accept / Check / Stop

The main idea is that different tasks should not necessarily need the same amount of verification.

For example:

Easy task
↓
Light checking
↓
Accept

But:

Difficult / uncertain / risky task
↓
More checking
↓
Strong verification
↓
Accept or stop
The components
AI**

The main AI produces a candidate answer.

EM — Evidence Module

EM produces evidence that can be checked.

One thing I want to investigate is whether EM should generate its evidence independently from the AI candidate.

ER — Evidence Reasoner

ER looks at the candidate and the evidence.

Its job is not simply to ask whether two outputs are similar. It should investigate whether the evidence actually supports the candidate.

Possible decisions are:

SUPPORTED
CHECK
INSUFFICIENT_EVIDENCE
AIC

AIC is a stronger checking stage.

The idea is that it should not necessarily be used for every input.

Why adaptive checking?

A very strong verifier could be used for every answer, but that may be unnecessary for simple tasks.

I want to investigate whether a system can start with a cheaper check and only use stronger verification when the earlier evidence is not sufficient.

So the idea is:

small problem → small check

uncertain problem → stronger check

high-risk problem → strongest check

The exact way to decide when to escalate is still an open research question.

What I am NOT claiming

I am not claiming that:

verification is a new research field,
uncertainty estimation is new,
self-verification is new,
evidence checking is new,
or that Project Reliance has already been proven to work.

There is already significant research in these areas.

I am interested in whether the combination of independent evidence + evidence reasoning + adaptive escalation is useful enough to study further.

Research question

The main question I want to test is:

Can adaptive evidence verification improve the reliability of AI systems while avoiding unnecessary strong verification on easy cases?

I would eventually like to compare:

**AI alone

AI + fixed verification

AI + strong verification every time

AI + adaptive verification**

and measure things such as:

accuracy
errors detected
false accepts
false rejects
verification cost
number of strong-checker calls
Why I am asking the PyTorch community

I am still at the design stage.

I would like to know:

Does this architecture make sense?
Does something similar already exist in the PyTorch ecosystem?
Would components such as EM and ER make sense as PyTorch modules?
What would be a good way to benchmark this idea?
Are there existing research directions I should study before continuing?

I am not asking for this to be added to PyTorch right now.

I mainly want feedback on whether the idea is technically meaningful and worth developing further.

Thanks for reading.

Start by reviewing the proposed AI, Evidence Module, Evidence Reasoner, and AIC architecture, then examine the open questions about adaptive escalation and benchmarking. Done would require a concrete implementation and comparison of AI alone, fixed verification, always-strong verification, and adaptive verification using accuracy, detected errors, false accepts, false rejects, cost, and strong-checker calls.