django-haystack / django-haystack/django-haystack

Custom analyzer with edgeNgram filter doesn't work.

Open
#1,563 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.7k
Forks
1.3k
Avg merge
3h 18m
Merged PRs (30d)
3

Description

Hi,

I needed partial search in my website.Initially I used edgeNgramFeild directly it didn't work as expected.So I used custom search engine with custom analyzers.

'settings': {
       "analysis": {
           "analyzer": {
               "ngram_analyzer": {
                   "type": "custom",
                   "tokenizer": "lowercase",
                   "filter": ["haystack_ngram"]
               },
               "edgengram_analyzer": {
                   "type": "custom",
                   "tokenizer": "lowercase",
                   "filter": ["haystack_edgengram"]
               },
               "suggest_analyzer": {
                   "type":"custom",
                   "tokenizer":"standard",
                   "filter":[
                       "standard",
                       "lowercase",
                       "asciifolding"
                   ]
               },
           },
           "tokenizer": {
               "haystack_ngram_tokenizer": {
                   "type": "nGram",
                   "min_gram": 3,
                   "max_gram": 15,
               },
               "haystack_edgengram_tokenizer": {
                   "type": "edgeNGram",
                   "min_gram": 2,
                   "max_gram": 15,
                   "side": "front"
               }
           },
           "filter": {
               "haystack_ngram": {
                   "type": "nGram",
                   "min_gram": 3,
                   "max_gram": 15
               },
               "haystack_edgengram": {
                   "type": "edgeNGram",
                   "min_gram": 2,
                   "max_gram": 15
               }
           }
       }
   } 

Used edgengram_analyzer for indexing and suggest_analyzer for seacrh. This worked for some extent.But,it doesn't work for numbers for example when 30 is entered it doesn't search for 303 and also with words containing alphabet and numbers combined. so I searched for various sites. They suggestsed to use standard or whitespace tokenizer and with haystack_edgengram filter.But it didn't work at all,puting aside number partial search didn't work even for alphabet.The settings after the suggestion:

'settings': {
        "analysis": {
            "analyzer": {
                "ngram_analyzer": {
                    "type": "custom",
                    "tokenizer": "lowercase",
                    "filter": ["haystack_ngram"]
                },
                "edgengram_analyzer": {
                    "type": "custom",
                    "tokenizer": "whitepsace",
                    "filter": ["haystack_edgengram"]
                },
                "suggest_analyzer": {
                    "type":"custom",
                    "tokenizer":"standard",
                    "filter":[
                        "standard",
                        "lowercase",
                        "asciifolding"
                    ]
                },
            },
            "filter": {
                "haystack_ngram": {
                    "type": "nGram",
                    "min_gram": 3,
                    "max_gram": 15
                },
                "haystack_edgengram": {
                    "type": "edgeNGram",
                    "min_gram": 2,
                    "max_gram": 15
                }
            }
        }
    } 

Does anything other than lowercase tokenizer work? or haystack_edgengram filter not working for me.According my knowlege it should work like this. Consideing 2 Lazy Dog as text supplied. it should get tokens like this with whitespace [2,Lazy,Dog]. and then applying haystack_edgengram filter it should generate tokens [2,la,laz,lazy,do,dog] .its not working like this.Did i do something wrong?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the analyzer settings and token examples in the issue, checking how the tokenizer and haystack_edgengram filter handle numeric and alphanumeric input. Verify the generated tokens for the stated examples; done means partial searches work for both numbers and words.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.