Avijit-Kumar-GIT / Avijit-Kumar-GIT/fella
verify: cost-gated self-consistency re-check for cases the shadow query can't cover
- Dominant language
- Rust
- Stars
- 0
- Forks
- 0
- Avg merge
- 5h 19m
- Merged PRs (30d)
- 72
Description
The fallback tier for #67-class value errors once the deterministic checks (shadow query, lexical mismatch — see related issues) don't apply: joins, multi-step, or anything outside the simple single-table shape.
**Mechanism.** When a soft-warning signal already fired (from any of the cheap checks) *and* the case doesn't qualify for a deterministic shadow query, spend one extra LLM round-trip: re-ask with a stricter, more literal prompt ("answer using only run_sql, be exact about every filter word in the question") and compare the two answers. Disagreement → surface as low-confidence (ties into the "surface low-confidence answers" issue) rather than silently picking one.
**Cost discipline is the point of this issue:** gate it behind an existing risk signal, never run it on every answer — an extra LLM call on every question would undo the "≤2 round trips" cost/latency target. This is the expensive tier, used sparingly, per the "targeted verification, not universal" pattern from the agent-reliability literature.
Related: #67, and whichever "surface low-confidence" issue is filed alongside this batch.
Contributor guide
Research direction
Read the related #67 issue and the issues for the cheap checks and surfacing low-confidence answers; the payload does not name files or tests. Trace where soft-warning signals are raised and where deterministic shadow-query eligibility is decided. Done means the extra, stricter LLM call runs only after a warning when a shadow query cannot cover the case, with disagreements surfaced as low-confidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, rust, sql
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100