Google Crawler can only get around 100 images instead of 1000
- Dominant language
- Python
- Stars
- 933
- Forks
- 181
- PR merge metrics
- No merged PRs in 30d
Description
Hi, when I used the searching URLs generated by `feed()` function in `GoogleFeeder`, I can only get around 100 images although the `max_num=1000`. I find that all the URLs get the same 100 results as the first URL. It seems that the `ijn` and `start` params are no use anymore. I just want to get nearly 1000 images per keyword. Is there anybody who has a solution?
```
def feed(self, keyword, offset, max_num, language=None, filters=None):
base_url = 'https://www.google.com/search?'
self.filter = self.get_filter()
filter_str = self.filter.apply(filters, sep=',')
for i in range(offset, offset + max_num, 100):
params = dict(
q=keyword,
ijn=int(i / 100),
start=i,
tbs=filter_str,
tbm='isch')
if language:
params['lr'] = 'lang_' + language
url = base_url + urlencode(params)
self.out_queue.put({'url': url, 'keyword': keyword, 'next_offset': i+100})
self.logger.debug('put url to url_queue: {}'.format(url))
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.