catboost / catboost/benchmarks

issue in the function computing NDCG

Open
#19 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
172
Forks
45
PR merge metrics
No merged PRs in 30d

Description

Hi,

I just find there is an issue in the following function
```
def ndcg(y_pred, y_true, top):
assert y_pred.shape[0] == y_true.shape[0]
top = min(top, y_pred.shape[0])

first_k_docs = sorted(zip(y_true, y_pred), key=cmp_to_key(doc_comparator))
first_k_docs = np.array(first_k_docs)[:top,0]

top_k_idxs = np.argsort(y_true)[::-1][:top]
top_k_docs = y_true[top_k_idxs]

dcg = cumulative_gain(first_k_docs)
idcg = cumulative_gain(top_k_docs)

return dcg / idcg if idcg > 0 else 1.
```

how can ndcg=1 if idcg == 0? If idcg == 0 you should just ignore that query. This definitely makes the NDCG look higher than it is expected to be.

Best,

Ruocheng Guo

Contributor guide

Open the contributing guide

Research direction

Start with the ndcg(y_pred, y_true, top) function shown in the issue and locate its definition in the benchmark code. Trace how queries with idcg == 0 are included in the aggregate result; done means those queries are ignored rather than contributing an NDCG of 1, with the affected metric behavior verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.