django-haystack / django-haystack/django-haystack

Configurable auto_generate_phrase_queries for Elasticsearch backend

Open
#1,325 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backend : elasticsearch needs review query
Dominant language
Python
Stars
3.7k
Forks
1.3k
Avg merge
3h 18m
Merged PRs (30d)
3

Description

  • [X ] Tested with the latest Haystack release
  • [X ] Tested with the current Haystack master branch

Expected behaviour

Haystack probably needs to support configuration of the auto_generate_phrase_queries parameter, which is directly send to Elasticsearch.

In some specific cases, for non-whitespace delimited phrases the returned results can be weird.

Actual behaviour

EdgeNGram field is not returning autocomplete suggestions in inappropriate order. Results that are obviously more natural to autosuggest are with the same score as items that are far away from natural

After digging into the backend, the following line of code was found to cause the problem:

https://github.com/django-haystack/django-haystack/blob/master/haystack/backends/elasticsearch_backend.py#L278

Probably that parameter needs to be at least configurable.

After checking what that parameter does, the following answer at StackOverflow helped to understand the whole picture.
http://stackoverflow.com/a/28694630

SOLR-2015: Add a boolean attribute autoGeneratePhraseQueries to TextField. autoGeneratePhraseQueries="true" (the default) causes the query parser to generate phrase queries if multiple tokens are generated from a single non-quoted analysis string. For example WordDelimiterFilter splitting text:pdp-11 will cause the parser to generate text:"pdp 11" rather than (text:PDP OR text:11). Note that autoGeneratePhraseQueries="true" tends to not work well for non whitespace delimited languages.

Steps to reproduce the behaviour

  1. Prepopulate data into an EdgeNGram field, containing data like "Title 03-000908-01P", "Title 03-000710-01P", "Bla 03-000866-01P" and similar items (which can be tokenized by "-", but believe me I have tried any other filters, analyzers, tokenizers, and a probably everything else)
  2. Perform a search using SearchQuerySet().auto_query(query, fieldname="title_auto") or in some similar manner
  3. When search is performed with that option set to true, and autocomplete query set to "03-0009", the first suggested result is "Bla 03-000866-01P".

Trying to explain a very specific case. The issue is reproducible in many other cases, as we are working with data containing a lot of punctual characters.

Configuration

  • Operating system version: Ubuntu 14.04
  • Search engine version:
 "version" : {
    "number" : "1.6.2",
    "build_hash" : "NA",
    "build_timestamp" : "NA",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },

  • Python version: 2.7.9
  • Django version: 1.9
  • Haystack version: master

I hope that helps.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in haystack/backends/elasticsearch_backend.py around the linked line 278, then trace how SearchQuerySet().auto_query() reaches the Elasticsearch backend. Reproduce the EdgeNGram example with the documented auto_generate_phrase_queries setting; done means that parameter can be configured and the Elasticsearch query path honors the selected value.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, elasticsearch, python
Domain
backend, search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.