jorgebastida / jorgebastida/awslogs

Getting throttled trying to search large number of streams -- with or without stream filters

Open
#103 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5k
Forks
342
PR merge metrics
No merged PRs in 30d

Description

$ awslogs get api-log --start='14h ago' --filter-pattern="1764_0019_03"
Traceback (most recent call last):
File "/usr/local/bin/awslogs", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/awslogs/bin.py", line 166, in main
getattr(logs, options.func)()
File "/usr/local/lib/python2.7/site-packages/awslogs/core.py", line 187, in list_logs
consumer()
File "/usr/local/lib/python2.7/site-packages/awslogs/core.py", line 136, in consumer
for event in generator():
File "/usr/local/lib/python2.7/site-packages/awslogs/core.py", line 123, in generator
response = self.client.filter_log_events(**kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the FilterLogEvents operation (reached max retries: 4): Rate exceeded

# There are a 115 of these.
$ awslogs get api-log chef-worker-cubes-2017* --start='14h ago' --filter-pattern="1764_0019_03"
The number of streams that match your pattern 'chef-worker-cubes-2017*' is '115'. AWS API limits the number of streams you can filter by to 100.It might be helpful to you to not filter streams by any pattern and filter the output of awslogs.

# There should only be 10 of these
$ awslogs get api-log chef-worker-cubes-20170101-0* --start='14h ago' --filter-pattern="1764_0019_03"
The number of streams that match your pattern 'chef-worker-cubes-20170101-0*' is '112'. AWS API limits the number of streams you can filter by to 100.It might be helpful to you to not filter streams by any pattern and filter the output of awslogs.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading awslogs/core.py, especially list_logs, consumer, generator, and the filter_log_events call shown in the traceback. Reproduce the search with more than 100 matching streams and compare it with the narrower pattern. Done means large-stream searches no longer fail with throttling or the 100-stream API limit, with behavior covered by an appropriate test if the repository provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.