openzim / openzim/libzim

Increase relevance of search results within ZIM by refining the indexed content

Open
#952 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
256
Forks
75
Avg merge
8d 51m
Merged PRs (30d)
4

Description

The FTS in xapian currently uses the entire text with accents removed in each webpage, but actually not all of it is relevant and this leads to suboptimal search results when searching inside the ZIM. On this, there's this library called readability that has an algorithm for scoring all DOM nodes and selecting the one where the actual webpage content is stored, or rather, is most likely to be stored (what is meant by "actual" is not things like "Similar products" or "See also" nodes). So it would be possible to only index the relevant content or even replace the original content with what readability believes is the relevant content. But either way the search results would improve as a result of this.

It would even be possible to have both indexes side-by-side, meaning, the index as it is right now, unchanged, and a new one with the proposed method.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/writer/defaultIndexData.h, especially the current use of each webpage's full text for the Xapian FTS index. Review Mozilla Readability's node-scoring algorithm in Readability.js and compare the options described: indexing only relevant content, replacing the indexed content, or keeping both indexes. Done means the chosen approach improves search relevance inside the ZIM.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.