althonos / althonos/InstaLooter

ConnectionResetError: [Errno 104] Connection reset by peer

未关闭
#278 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
bug
主要语言
Python
星标
2.1k
派生
257
PR 合并指标
30 天内没有已合并 PR

描述

## Environment

*Describe here your environment, including:*

* *Ubuntu*
* *Python 3.6*

## Error description - runtime

*I'm using InstaLooter as a python module. I have a script where you can specify date range and hashtag. Then I use Instalooter.Hashtaglooter object and iterate through pages to extract only posts where date is within given date range (I download created dictionary with only info I need and save as json, then I download by requests library image from "display_urls" in post)*

### Reproducible test case

*Whole python script is on Azure VM.
Case is interesting. When I run described script Connection Reset Error is raised. Same situation is when I try to download 400 posts and 200 posts, always somewhere at the end of the process.

I created another VM and there I can download as many post from insta with given hashtag as I want using CLI command (I cannot use date range in this way, thats why I created customized python script). *

```python
def scrape_instagram(hashtag="", begin_date="", end_date="", output_directory=""):

insta_posts_directory = output_directory + r"\posts"
insta_pictures_directory = output_directory + r"\img"

begin_date = begin_date.split("-")
begin_date_year = int(begin_date[0])
begin_date_month = int(begin_date[1])
begin_date_day = int(begin_date[2])

end_date = end_date.split("-")
end_date_year = int(end_date[0])
end_date_month = int(end_date[1])
end_date_day = int(end_date[2])

begin = datetime.datetime(end_date_year, end_date_month, end_date_day)
end = datetime.datetime(begin_date_year, begin_date_month, begin_date_day)

dictionary_number = {} # Dict for every single day of given time range and number of downloaded posts
dictionary_values = {} # Dict for boolean value restricts if day needs download more posts

date_difference = begin - end
number_of_days = int(date_difference.days)
for i in range(1, number_of_days+2):
dictionary_number[i] = 0
dictionary_values[i] = True

# Create object of HashtagLooter class
scraper = HashtagLooter(hashtag=hashtag, dump_json=True, dump_only=True, extended_dump=True)
pages = scraper.pages()

# Iterate through returned pages
for page in pages:
posts = page['edge_hashtag_to_media']['edges']

for post in posts:
date = datetime.datetime.fromtimestamp(post['node']['taken_at_timestamp']) # Read date from given post

if True in dictionary_values.values(): # Check if there are still days with need to downloaded posts

if begin > date > end: # Check if date of post is in given range

if len(post['node']['edge_media_to_caption']['edges']) > 0: # Check if post has a "text" parameter
day = date.day

if dictionary_values[day]: # Check if day taken from post need to download more posts
dictionary_number[day] += 1

to_save = {} # Create dictionary to save as json afterwards
post_id = post['node']['id']
to_save['post_id'] = post_id
to_save['text'] = post['node']['edge_media_to_caption']['edges'][0]['node']['text']
to_save['edge_media_to_comment'] = post['node']['edge_media_to_comment']['count']
display_url = post['node']['display_url']
to_save['display_url'] = display_url
to_save['edge_liked_by'] = post['node']['edge_liked_by']['count']
to_save['taken_at_timestamp'] = post['node']['taken_at_timestamp']

with open(insta_posts_directory + "\\" + post_id + ".json", 'w') as file: # Save json
json.dump(to_save, file)

img_data = requests.get(display_url).content
with open(insta_pictures_directory + '\\' + post_id + '.jpg', 'wb') as handler: # Download img
handler.write(img_data)

if dictionary_number[day] == 300:
dictionary_values[day] = False
else:
break
```

*Do you have any idea why this error is raised and how I can manage this issue?*

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。