Inqui-Lab: Improving duplicate-detection

Context

Input app pilot testing (~2,000 dataset) surfaced missing duplicate detection, a desire for a global knowledge base, and duplicate detection linked to topic relevance. These are mostly prompt + pipeline changes.

Approach

Option 1:

• Keep duplicate detection independent of L2 scoring; handle it in post-processing via a weighted average (a flagged idea may still be strong relative to the cohort — a decision input, not a score penalty).
• Duplicate detection is ~40–50% accurate — communicate plainly, keep human-in-loop.
• Cheaper alternative to global web search: lean on the model's trained knowledge via rubric wording ("if the idea is generally common, score lower on novelty").
  Option 2:
• Feed the duplicate-detection result as an input into the next LLM stage (not done today).
• Prompt then instructs the LLM to consider the duplicate result and assign appropriate novelty scores.
• Need the Input app team's help on how they want the prompt changes shaped.
• Keep some scope for Google search but treat cautiously (nondeterministic wide search).