django-haystack / django-haystack/django-haystack

narrowed_results limit causing search results to be removed (Whoosh)

Open
#351 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs review
Dominant language
Python
Stars
3.7k
Forks
1.3k
Avg merge
3h 18m
Merged PRs (30d)
3

Description

I'm running into an issue where the narrowed_results result set is causing valid raw_results to be dropped. Since HAYSTACK_LIMIT_TO_REGISTERED_MODELS is set to True, the generated narrow query is:

django_ct:(flatpages.flatpage OR topics.topic)

When this query is run by narrow_searcher, it returns a result set of:

<Top 10 Results for Or([Term(u'django_ct', u'flatpages.flatpage'), Term(u'django_ct', u'topics.topic')]) runtime=0.0010199546814>

Since the result set is limited to 10 results, the call to raw_results.filter(narrowed_results) is filtering out all sorts of results from raw_results that are actually valid, but just happen to not be in the first 10 results of narrowed_results. By changing line 316 in whoosh_backend.py from:

recent_narrowed_results = narrow_searcher.search(self.parser.parse(force_unicode(nq)))

to

recent_narrowed_results = narrow_searcher.search(self.parser.parse(force_unicode(nq)), limit=None)

the correct results are now being returned. This could be quite expensive, so I'm not sure how you'd want to handle this case. My short term solution is going to be setting HAYSTACK_LIMIT_TO_REGISTERED_MODELS to False.

I'm using the Whoosh backend with django-haystack 1.2.0 and whoosh 1.8.2.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in whoosh_backend.py around line 316 and inspect how narrow_searcher.search results are passed to raw_results.filter. Reproduce the registered-model narrowing case described in the issue, then ensure valid raw_results are not removed merely because they fall outside the narrowed result limit while considering the cost of an unlimited search.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.