catboost / catboost/benchmarks
issue in the function computing NDCG
- Dominant language
- Jupyter Notebook
- Stars
- 172
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I just find there is an issue in the following function
```
def ndcg(y_pred, y_true, top):
assert y_pred.shape[0] == y_true.shape[0]
top = min(top, y_pred.shape[0])
first_k_docs = sorted(zip(y_true, y_pred), key=cmp_to_key(doc_comparator))
first_k_docs = np.array(first_k_docs)[:top,0]
top_k_idxs = np.argsort(y_true)[::-1][:top]
top_k_docs = y_true[top_k_idxs]
dcg = cumulative_gain(first_k_docs)
idcg = cumulative_gain(top_k_docs)
return dcg / idcg if idcg > 0 else 1.
```
how can ndcg=1 if idcg == 0? If idcg == 0 you should just ignore that query. This definitely makes the NDCG look higher than it is expected to be.
Best,
Ruocheng Guo
Contributor guide
Research direction
Start with the ndcg(y_pred, y_true, top) function shown in the issue and locate its definition in the benchmark code. Trace how queries with idcg == 0 are included in the aggregate result; done means those queries are ignored rather than contributing an NDCG of 1, with the affected metric behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100