jorgebastida / jorgebastida/awslogs
Connection Reset while fetching logs
- Dominant language
- Python
- Stars
- 5k
- Forks
- 342
- PR merge metrics
- No merged PRs in 30d
Description
FYI: The stream I am trying to fetch has lot's of logs.
Got the following error output:
```
Version: 0.10.0
Python: 3.6.2 (default, Sep 30 2017, 02:25:46)
[GCC 5.4.0 20160609]
boto3 version: 1.4.7
Platform: Linux-4.10.0-38-generic-x86_64-with-debian-stretch-sid
Config: {'aws_access_key_id': 'SENSITIVE', 'aws_secret_access_key': 'SENSITIVE', 'aws_session_token': 'SENSITIVE', 'aws_profile': 'SENSITIVE', 'aws_region': None, 'log_group_name': '', 'log_stream_name': '', 'filter_pattern': None, 'watch': False, 'output_group_enabled': True, 'output_stream_enabled': True, 'output_timestamp_enabled': False, 'output_ingestion_time_enabled': False, 'start': '2d ago', 'end': None, 'color_enabled': True, 'query': None, 'func': 'list_logs'}
Args: ['/home/prayashm/.pyenv/versions/3.6.2/bin/awslogs', 'get', '', '', '--profile=', '--start=2d ago']
Traceback (most recent call last):
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/awslogs/bin.py", line 172, in main
getattr(logs, options.func)()
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/awslogs/core.py", line 196, in list_logs
consumer()
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/awslogs/core.py", line 137, in consumer
for event in generator():
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/awslogs/core.py", line 124, in generator
response = self.client.filter_log_events(**kwargs)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/client.py", line 312, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/client.py", line 592, in _make_api_call
operation_model, request_dict)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/endpoint.py", line 249, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/endpoint.py", line 204, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 605, in send
r.content
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/vendored/requests/models.py", line 750, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/vendored/requests/models.py", line 673, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 307, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 243, in read
data = self._fp.read(amt)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 1002, in recv_into
return self.read(nbytes, buffer)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 865, in read
return self._sslobj.read(len, buffer)
File "/home/prayashm/.pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 625, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in awslogs/core.py around list_logs, consumer, generator, and the filter_log_events call shown in the traceback. Reproduce the failure while fetching a large stream over the reported two-day range, then trace how the connection reset is handled. Done means the command can fetch large streams without terminating with ConnectionResetError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100