SPIKE: Investigate Reranker Minimization Threshold

Background

TTC relies on a retriever to identify candidate matches and a reranker to select the best standardization result. Initial analysis suggests that the retriever performs well, while the reranker struggles in some scenarios.

This behavior may not be solely attributable to reranker quality. Rerankers are generally most effective when making nuanced distinctions between multiple plausible candidates. They are less effective when:

• The correct answer is already obvious from retriever outputs.
• All retrieved candidates are poor matches.
• The retriever returns candidates with uniformly low similarity scores, making it difficult for the reranker to identify meaningful signal.

One potential optimization is pruning the results sent to the reranker to minimize the number of poor matches. The retriever currently returns the top-k nearest neighbors, all of which are passed to the reranker. However, lower-ranked neighbors often have substantially lower cosine similarity scores than the leading candidates and may contribute little value to the reranking process.

For example, if the retriever returns several candidates with similarity scores above 0.7 followed by candidates below 0.4, the lower-scoring candidates may be unlikely to influence the final ranking. Removing these candidates before reranking could reduce reranker workload and latency while preserving prediction quality.

This spike will investigate whether low-scoring retrieved candidates can be safely excluded from reranking and identify a cutoff point (margin) beyond which candidates are unlikely to contribute meaningful information to the reranker. The goal is to determine whether candidate sets can be reduced by pruning candidates that fall beyond significant score drops while preserving reranker behavior.

Acceptance Criteria

• Margin Distribution analysis

  • Calculate score differences between adjacent retriever candidates for the top-k retrieval results, e.g., rank 1-2, 2-3, etc.
    • Create visualizations showing the distribution of adjacent-candidate margins. (I don't know if this will actually show anything useful...)
  • Quantify how frequently large score drops occur within the top-k results.
  • Identify common retrieval patterns, e.g., clear candidate clusters followed by large score drops or gradually declining score distributions with no obvious cutoff.

• Candidate Set Boundary Analysis

  • Quantify the percentage, count, and average remaining candidates under various candidate margin pruning thresholds (e.g., ≥0.05, ≥0.1, ≥0.15).
  • Create visualizations to show above

• Reranker Impact Analysis

  • Evaluate how often margin-based pruning changes the reranker's final selected candidate relative to the baseline top-k approach. Note: this should probably exclude auto-acceptances if we find strong evidence for adding that threshold.
  • Quantify the percentage of inputs whose final prediction remains unchanged after pruning.
  • Manual inspection: identify representative examples where pruning alters the final prediction.

• Produce a recommendation deck for APHL summarizing the analysis, proposed theshold(s) if applicable, rationale, and expect impact on coverage and prediction quality (with concrete examples if possible)

  • Determine whether there is evidence supporting margin-based pruning and at what margin if applicable