bisohns / bisohns/search-engine-parser

Bing search is broken

Open
#171 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
491
Forks
87
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Running simple code (based on the Readme)

getting:
```python
ENGINE FAILURE: Bing
Traceback (most recent call last):
File "/home/bentsi/pycharm-community-2021.2/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/bentsi/pycharm-community-2021.2/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/bentsi/devel/continueai/backend/src/scraping/search_engine_query.py", line 17, in
bresults = bsearch.search(**search_args)
File "/home/bentsi/.pyenv/versions/cai-backend/lib/python3.10/site-packages/search_engine_parser/core/base.py", line 288, in search
return self.get_results(soup, **kwargs)
File "/home/bentsi/.pyenv/versions/cai-backend/lib/python3.10/site-packages/search_engine_parser/core/base.py", line 247, in get_results
raise NoResultsOrTrafficError(
search_engine_parser.core.exceptions.NoResultsOrTrafficError: The result parsing was unsuccessful. It is either your query could not be found or it was flagged as unusual traffic
```
after digging into the root cause I found following:
1) http request to Bing returns response with HTML without results
![image](https://user-images.githubusercontent.com/5269241/178799590-0d44fc7a-4a8f-4877-b489-1da39061cde3.png)
2) after adding a cookie that Google Chrome adds to GET headers, the code starts working
![image](https://user-images.githubusercontent.com/5269241/178800110-7ecd97d9-04a6-4b86-ae98-12d4b1515fd7.png)

So the solution is to add cookie data, but I am not sure what exactly should be added, since cookie looks sophisticated.

**To Reproduce**
```
from search_engine_parser.core.engines.bing import Search as BingSearch
company_name = "samsung electronics corp official website"

search_args = {"query": company_name, "page": 1}
bsearch = BingSearch()
bsearch.clear_cache()
bresults = bsearch.search(**search_args)
```
**Expected behavior**
Search returns results
**Screenshots**

**Desktop (please complete the following information):**
- OS: Ubuntu 20.04
- Python Version: 3.10.5
- Search-engine-parser version: 0.6.6

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.