Increase relevance of search results within ZIM by refining the indexed content
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 256
- Forks
- 75
- Avg merge
- 8d 51m
- Merged PRs (30d)
- 4
Description
The FTS in xapian currently uses the entire text with accents removed in each webpage, but actually not all of it is relevant and this leads to suboptimal search results when searching inside the ZIM. On this, there's this library called readability that has an algorithm for scoring all DOM nodes and selecting the one where the actual webpage content is stored, or rather, is most likely to be stored (what is meant by "actual" is not things like "Similar products" or "See also" nodes). So it would be possible to only index the relevant content or even replace the original content with what readability believes is the relevant content. But either way the search results would improve as a result of this.
It would even be possible to have both indexes side-by-side, meaning, the index as it is right now, unchanged, and a new one with the proposed method.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/writer/defaultIndexData.h, especially the current use of each webpage's full text for the Xapian FTS index. Review Mozilla Readability's node-scoring algorithm in Readability.js and compare the options described: indexing only relevant content, replacing the indexed content, or keeping both indexes. Done means the chosen approach improves search relevance inside the ZIM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100