Bug: ServiceException: An error occurred (ServiceException) when calling the Invoke operation
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
Observed crash in apps function related to invoking the rule processor:
```
Traceback (most recent call last):
File "/var/task/app_integrations/main.py", line 39, in handler
app.gather()
File "/var/task/app_integrations/apps/app_base.py", line 438, in gather
while (self._gather() + self._sleep_seconds() <
File "/var/task/app_integrations/apps/app_base.py", line 423, in _gather
exec_time = Decimal(timeit(do_gather, number=1))
File "/usr/lib64/python2.7/timeit.py", line 237, in timeit
return Timer(stmt, setup, timer).timeit(number)
File "/usr/lib64/python2.7/timeit.py", line 202, in timeit
timing = self.inner(it, self.timer)
File "/usr/lib64/python2.7/timeit.py", line 100, in inner
_func()
File "/var/task/app_integrations/apps/app_base.py", line 412, in do_gather
self._batcher.send_logs(self._config['function_name'], logs)
File "/var/task/app_integrations/batcher.py", line 53, in send_logs
if self._send_logs_to_stream_alert(source_function, logs):
File "/var/task/app_integrations/batcher.py", line 114, in _send_logs_to_stream_alert
Qualifier='production'
File "/var/runtime/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
ServiceException: An error occurred (ServiceException) when calling the Invoke operation (reached max retries: 4): An error occurred and the request cannot be processed.
```
## Desired Change
Investigate what's going on and probably not cause a hard crash :)
Contributor guide
Research direction
Start with app_integrations/batcher.py, especially _send_logs_to_stream_alert, and follow its invocation from app_integrations/apps/app_base.py. Investigate the AWS Invoke ServiceException and retry exhaustion shown in the traceback. Done means the failure is understood and an invocation error no longer causes the app to hard-crash, with the relevant behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100