althonos / althonos/InstaLooter

ConnectionResetError: [Errno 104] Connection reset by peer

Abierto
#278 0 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
2.1k
Forks
257
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Environment

*Describe here your environment, including:*

* *Ubuntu*
* *Python 3.6*

## Error description - runtime

*I'm using InstaLooter as a python module. I have a script where you can specify date range and hashtag. Then I use Instalooter.Hashtaglooter object and iterate through pages to extract only posts where date is within given date range (I download created dictionary with only info I need and save as json, then I download by requests library image from "display_urls" in post)*

### Reproducible test case

*Whole python script is on Azure VM.
Case is interesting. When I run described script Connection Reset Error is raised. Same situation is when I try to download 400 posts and 200 posts, always somewhere at the end of the process.

I created another VM and there I can download as many post from insta with given hashtag as I want using CLI command (I cannot use date range in this way, thats why I created customized python script). *

```python
def scrape_instagram(hashtag="", begin_date="", end_date="", output_directory=""):

insta_posts_directory = output_directory + r"\posts"
insta_pictures_directory = output_directory + r"\img"

begin_date = begin_date.split("-")
begin_date_year = int(begin_date[0])
begin_date_month = int(begin_date[1])
begin_date_day = int(begin_date[2])

end_date = end_date.split("-")
end_date_year = int(end_date[0])
end_date_month = int(end_date[1])
end_date_day = int(end_date[2])

begin = datetime.datetime(end_date_year, end_date_month, end_date_day)
end = datetime.datetime(begin_date_year, begin_date_month, begin_date_day)

dictionary_number = {} # Dict for every single day of given time range and number of downloaded posts
dictionary_values = {} # Dict for boolean value restricts if day needs download more posts

date_difference = begin - end
number_of_days = int(date_difference.days)
for i in range(1, number_of_days+2):
dictionary_number[i] = 0
dictionary_values[i] = True

# Create object of HashtagLooter class
scraper = HashtagLooter(hashtag=hashtag, dump_json=True, dump_only=True, extended_dump=True)
pages = scraper.pages()

# Iterate through returned pages
for page in pages:
posts = page['edge_hashtag_to_media']['edges']

for post in posts:
date = datetime.datetime.fromtimestamp(post['node']['taken_at_timestamp']) # Read date from given post

if True in dictionary_values.values(): # Check if there are still days with need to downloaded posts

if begin > date > end: # Check if date of post is in given range

if len(post['node']['edge_media_to_caption']['edges']) > 0: # Check if post has a "text" parameter
day = date.day

if dictionary_values[day]: # Check if day taken from post need to download more posts
dictionary_number[day] += 1

to_save = {} # Create dictionary to save as json afterwards
post_id = post['node']['id']
to_save['post_id'] = post_id
to_save['text'] = post['node']['edge_media_to_caption']['edges'][0]['node']['text']
to_save['edge_media_to_comment'] = post['node']['edge_media_to_comment']['count']
display_url = post['node']['display_url']
to_save['display_url'] = display_url
to_save['edge_liked_by'] = post['node']['edge_liked_by']['count']
to_save['taken_at_timestamp'] = post['node']['taken_at_timestamp']

with open(insta_posts_directory + "\\" + post_id + ".json", 'w') as file: # Save json
json.dump(to_save, file)

img_data = requests.get(display_url).content
with open(insta_pictures_directory + '\\' + post_id + '.jpg', 'wb') as handler: # Download img
handler.write(img_data)

if dictionary_number[day] == 300:
dictionary_values[day] = False
else:
break
```

*Do you have any idea why this error is raised and how I can manage this issue?*

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.