django-haystack / django-haystack/django-haystack
Custom analyzer with edgeNgram filter doesn't work.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.7k
- Forks
- 1.3k
- Avg merge
- 3h 18m
- Merged PRs (30d)
- 3
Description
Hi,
I needed partial search in my website.Initially I used edgeNgramFeild directly it didn't work as expected.So I used custom search engine with custom analyzers.
'settings': {
"analysis": {
"analyzer": {
"ngram_analyzer": {
"type": "custom",
"tokenizer": "lowercase",
"filter": ["haystack_ngram"]
},
"edgengram_analyzer": {
"type": "custom",
"tokenizer": "lowercase",
"filter": ["haystack_edgengram"]
},
"suggest_analyzer": {
"type":"custom",
"tokenizer":"standard",
"filter":[
"standard",
"lowercase",
"asciifolding"
]
},
},
"tokenizer": {
"haystack_ngram_tokenizer": {
"type": "nGram",
"min_gram": 3,
"max_gram": 15,
},
"haystack_edgengram_tokenizer": {
"type": "edgeNGram",
"min_gram": 2,
"max_gram": 15,
"side": "front"
}
},
"filter": {
"haystack_ngram": {
"type": "nGram",
"min_gram": 3,
"max_gram": 15
},
"haystack_edgengram": {
"type": "edgeNGram",
"min_gram": 2,
"max_gram": 15
}
}
}
}
Used edgengram_analyzer for indexing and suggest_analyzer for seacrh. This worked for some extent.But,it doesn't work for numbers for example when 30 is entered it doesn't search for 303 and also with words containing alphabet and numbers combined. so I searched for various sites. They suggestsed to use standard or whitespace tokenizer and with haystack_edgengram filter.But it didn't work at all,puting aside number partial search didn't work even for alphabet.The settings after the suggestion:
'settings': {
"analysis": {
"analyzer": {
"ngram_analyzer": {
"type": "custom",
"tokenizer": "lowercase",
"filter": ["haystack_ngram"]
},
"edgengram_analyzer": {
"type": "custom",
"tokenizer": "whitepsace",
"filter": ["haystack_edgengram"]
},
"suggest_analyzer": {
"type":"custom",
"tokenizer":"standard",
"filter":[
"standard",
"lowercase",
"asciifolding"
]
},
},
"filter": {
"haystack_ngram": {
"type": "nGram",
"min_gram": 3,
"max_gram": 15
},
"haystack_edgengram": {
"type": "edgeNGram",
"min_gram": 2,
"max_gram": 15
}
}
}
}
Does anything other than lowercase tokenizer work? or haystack_edgengram filter not working for me.According my knowlege it should work like this. Consideing 2 Lazy Dog as text supplied. it should get tokens like this with whitespace [2,Lazy,Dog]. and then applying haystack_edgengram filter it should generate tokens [2,la,laz,lazy,do,dog] .its not working like this.Did i do something wrong?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the analyzer settings and token examples in the issue, checking how the tokenizer and haystack_edgengram filter handle numeric and alphanumeric input. Verify the generated tokens for the stated examples; done means partial searches work for both numbers and words.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100