capreolus-ir / capreolus-ir/diffir

Add summary of metrics to landing page

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
20
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Based on earlier discussions, it would be nice to display the following on the landing page to help assess a single run:
1. Number of unjudged docs (in top K)
2. Metrics with upper and lower bounds (based on assumptions about unjudged docs)

And to assess pairs of runs:
3. Wins/losses/draws between the two runs (e.g., number of queries with a > 0.01 difference in a metric)

Contributor guide

No contributing guide indexed for this repository

Research direction

No implementation files or tests are identified in the issue. Start by locating the landing-page entry point and the code that computes or presents run metrics, then determine how single-run bounds and pairwise wins, losses, and draws should be represented. Done means the landing page displays all three requested summaries for one run and for a pair of runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.