internetarchive / internetarchive/openlibrary

Create dataset of examples of poor search edition ranking

Open
#11,694 9 comments 0 reactions 0 assignees View on GitHub
1-off tasks Lead: @cdrini Priority: 3 Theme: Search
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

Edition ranking is what edition (title, cover) is chosen for a given work in search results. E.g. is it the french edition? One with a cover? One that is readable? This issue is complete once we have a list of ~20-50 cases where the _wrong_ edition was selected for a certain query.

The current logic is as follows:
https://github.com/internetarchive/openlibrary/blob/09cbf7d9b2868abdf7abfbf9b0746751a747207d/openlibrary/plugins/worksearch/schemes/works.py#L522-L539

We have noticed various cases where this ranking appears to not be operating correctly. We hypothesize some reasons could be:
* the inclusion of stopwords in our index causing "the lord of the rings" and "lord of the rings" to rank differently
* the term frequency algorithm our solr is configured to use, ala https://github.com/internetarchive/openlibrary/issues/10969#issuecomment-3096849420
* `language`, other key metadata missing from the work records

In order to test improvements, we need a list of failure cases we can use to adjust the ranking algorithm, and then compare a before/after. We can build a control set by using validate popular book queries, common queries from our logs, and random queries from our logs (like in our previous [search evaluation](https://docs.google.com/spreadsheets/d/1BN5I7-OkTPaoTr2Es6jQ4O9ICWFmH0q9CP6kEgolCgg/edit#gid=1006480604)). But finding specific failure cases automatically is tricky, so we will collect failure cases on the issue.

### Stakeholders
@mekarpeles

Contributor guide

Open the contributing guide

Research direction

Read the edition-ranking logic in openlibrary/plugins/worksearch/schemes/works.py at lines 522-539, then review the linked search-evaluation spreadsheet and the suggested popular, logged, and random query sources. Collect approximately 20-50 cases where the wrong edition is selected for a query, recording each failure clearly so the set can be used to compare ranking changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.