Avijit-Kumar-GIT / Avijit-Kumar-GIT/fella

verify: cost-gated self-consistency re-check for cases the shadow query can't cover

Open
#80 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
0
Forks
0
Avg merge
5h 19m
Merged PRs (30d)
72

Description

The fallback tier for #67-class value errors once the deterministic checks (shadow query, lexical mismatch — see related issues) don't apply: joins, multi-step, or anything outside the simple single-table shape.

**Mechanism.** When a soft-warning signal already fired (from any of the cheap checks) *and* the case doesn't qualify for a deterministic shadow query, spend one extra LLM round-trip: re-ask with a stricter, more literal prompt ("answer using only run_sql, be exact about every filter word in the question") and compare the two answers. Disagreement → surface as low-confidence (ties into the "surface low-confidence answers" issue) rather than silently picking one.

**Cost discipline is the point of this issue:** gate it behind an existing risk signal, never run it on every answer — an extra LLM call on every question would undo the "≤2 round trips" cost/latency target. This is the expensive tier, used sparingly, per the "targeted verification, not universal" pattern from the agent-reliability literature.

Related: #67, and whichever "surface low-confidence" issue is filed alongside this batch.

Contributor guide

Open the contributing guide

Research direction

Read the related #67 issue and the issues for the cheap checks and surfacing low-confidence answers; the payload does not name files or tests. Trace where soft-warning signals are raised and where deterministic shadow-query eligibility is decided. Done means the extra, stricter LLM call runs only after a warning when a shadow query cannot cover the case, with disagreements surfaced as low-confidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, rust, sql
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.