althonos / althonos/InstaLooter
ConnectionResetError: [Errno 104] Connection reset by peer
- Lenguaje dominante
- Python
- Estrellas
- 2.1k
- Forks
- 257
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
## Environment
*Describe here your environment, including:*
* *Ubuntu*
* *Python 3.6*
## Error description - runtime
*I'm using InstaLooter as a python module. I have a script where you can specify date range and hashtag. Then I use Instalooter.Hashtaglooter object and iterate through pages to extract only posts where date is within given date range (I download created dictionary with only info I need and save as json, then I download by requests library image from "display_urls" in post)*
### Reproducible test case
*Whole python script is on Azure VM.
Case is interesting. When I run described script Connection Reset Error is raised. Same situation is when I try to download 400 posts and 200 posts, always somewhere at the end of the process.
I created another VM and there I can download as many post from insta with given hashtag as I want using CLI command (I cannot use date range in this way, thats why I created customized python script). *
```python
def scrape_instagram(hashtag="", begin_date="", end_date="", output_directory=""):
insta_posts_directory = output_directory + r"\posts"
insta_pictures_directory = output_directory + r"\img"
begin_date = begin_date.split("-")
begin_date_year = int(begin_date[0])
begin_date_month = int(begin_date[1])
begin_date_day = int(begin_date[2])
end_date = end_date.split("-")
end_date_year = int(end_date[0])
end_date_month = int(end_date[1])
end_date_day = int(end_date[2])
begin = datetime.datetime(end_date_year, end_date_month, end_date_day)
end = datetime.datetime(begin_date_year, begin_date_month, begin_date_day)
dictionary_number = {} # Dict for every single day of given time range and number of downloaded posts
dictionary_values = {} # Dict for boolean value restricts if day needs download more posts
date_difference = begin - end
number_of_days = int(date_difference.days)
for i in range(1, number_of_days+2):
dictionary_number[i] = 0
dictionary_values[i] = True
# Create object of HashtagLooter class
scraper = HashtagLooter(hashtag=hashtag, dump_json=True, dump_only=True, extended_dump=True)
pages = scraper.pages()
# Iterate through returned pages
for page in pages:
posts = page['edge_hashtag_to_media']['edges']
for post in posts:
date = datetime.datetime.fromtimestamp(post['node']['taken_at_timestamp']) # Read date from given post
if True in dictionary_values.values(): # Check if there are still days with need to downloaded posts
if begin > date > end: # Check if date of post is in given range
if len(post['node']['edge_media_to_caption']['edges']) > 0: # Check if post has a "text" parameter
day = date.day
if dictionary_values[day]: # Check if day taken from post need to download more posts
dictionary_number[day] += 1
to_save = {} # Create dictionary to save as json afterwards
post_id = post['node']['id']
to_save['post_id'] = post_id
to_save['text'] = post['node']['edge_media_to_caption']['edges'][0]['node']['text']
to_save['edge_media_to_comment'] = post['node']['edge_media_to_comment']['count']
display_url = post['node']['display_url']
to_save['display_url'] = display_url
to_save['edge_liked_by'] = post['node']['edge_liked_by']['count']
to_save['taken_at_timestamp'] = post['node']['taken_at_timestamp']
with open(insta_posts_directory + "\\" + post_id + ".json", 'w') as file: # Save json
json.dump(to_save, file)
img_data = requests.get(display_url).content
with open(insta_pictures_directory + '\\' + post_id + '.jpg', 'wb') as handler: # Download img
handler.write(img_data)
if dictionary_number[day] == 300:
dictionary_values[day] = False
else:
break
```
*Do you have any idea why this error is raised and how I can manage this issue?*
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.