airbnb / airbnb/streamalert

Bug: ServiceException: An error occurred (ServiceException) when calling the Invoke operation

Open
#633 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

## Background

Observed crash in apps function related to invoking the rule processor:
```
Traceback (most recent call last):
File "/var/task/app_integrations/main.py", line 39, in handler
app.gather()
File "/var/task/app_integrations/apps/app_base.py", line 438, in gather
while (self._gather() + self._sleep_seconds() <
File "/var/task/app_integrations/apps/app_base.py", line 423, in _gather
exec_time = Decimal(timeit(do_gather, number=1))
File "/usr/lib64/python2.7/timeit.py", line 237, in timeit
return Timer(stmt, setup, timer).timeit(number)
File "/usr/lib64/python2.7/timeit.py", line 202, in timeit
timing = self.inner(it, self.timer)
File "/usr/lib64/python2.7/timeit.py", line 100, in inner
_func()
File "/var/task/app_integrations/apps/app_base.py", line 412, in do_gather
self._batcher.send_logs(self._config['function_name'], logs)
File "/var/task/app_integrations/batcher.py", line 53, in send_logs
if self._send_logs_to_stream_alert(source_function, logs):
File "/var/task/app_integrations/batcher.py", line 114, in _send_logs_to_stream_alert
Qualifier='production'
File "/var/runtime/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
ServiceException: An error occurred (ServiceException) when calling the Invoke operation (reached max retries: 4): An error occurred and the request cannot be processed.
```

## Desired Change

Investigate what's going on and probably not cause a hard crash :)

Contributor guide

Open the contributing guide

Research direction

Start with app_integrations/batcher.py, especially _send_logs_to_stream_alert, and follow its invocation from app_integrations/apps/app_base.py. Investigate the AWS Invoke ServiceException and retry exhaustion shown in the traceback. Done means the failure is understood and an invocation error no longer causes the app to hard-crash, with the relevant behavior verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.