jorgebastida / jorgebastida/awslogs
Getting throttled trying to search large number of streams -- with or without stream filters
- Dominant language
- Python
- Stars
- 5k
- Forks
- 342
- PR merge metrics
- No merged PRs in 30d
Description
$ awslogs get api-log --start='14h ago' --filter-pattern="1764_0019_03"
Traceback (most recent call last):
File "/usr/local/bin/awslogs", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/awslogs/bin.py", line 166, in main
getattr(logs, options.func)()
File "/usr/local/lib/python2.7/site-packages/awslogs/core.py", line 187, in list_logs
consumer()
File "/usr/local/lib/python2.7/site-packages/awslogs/core.py", line 136, in consumer
for event in generator():
File "/usr/local/lib/python2.7/site-packages/awslogs/core.py", line 123, in generator
response = self.client.filter_log_events(**kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the FilterLogEvents operation (reached max retries: 4): Rate exceeded
# There are a 115 of these.
$ awslogs get api-log chef-worker-cubes-2017* --start='14h ago' --filter-pattern="1764_0019_03"
The number of streams that match your pattern 'chef-worker-cubes-2017*' is '115'. AWS API limits the number of streams you can filter by to 100.It might be helpful to you to not filter streams by any pattern and filter the output of awslogs.
# There should only be 10 of these
$ awslogs get api-log chef-worker-cubes-20170101-0* --start='14h ago' --filter-pattern="1764_0019_03"
The number of streams that match your pattern 'chef-worker-cubes-20170101-0*' is '112'. AWS API limits the number of streams you can filter by to 100.It might be helpful to you to not filter streams by any pattern and filter the output of awslogs.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading awslogs/core.py, especially list_logs, consumer, generator, and the filter_log_events call shown in the traceback. Reproduce the search with more than 100 matching streams and compare it with the narrower pattern. Done means large-stream searches no longer fail with throttling or the 100-stream API limit, with behavior covered by an appropriate test if the repository provides one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100