airbnb / airbnb/streamalert

[improvement] consolidate the kinesis_streams and kinesis_events terraform modules

Open
#908 0 comments 0 reactions 0 assignees View on GitHub
improvement terraform
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

## Background

There are currently two different terraform modules related to kinesis:

[tf_stream_alert_kinesis_events](https://github.com/airbnb/streamalert/tree/277d3b9664441feb2b2037055c101e54fb30c676/terraform/modules/tf_stream_alert_kinesis_events)
[tf_stream_alert_kinesis_streams](https://github.com/airbnb/streamalert/tree/277d3b9664441feb2b2037055c101e54fb30c676/terraform/modules/tf_stream_alert_kinesis_streams)

### Description

It is unclear why these should be separate, since they are very much so dependent on each other.. For example, the `kinesis_events` module cannot be enabled if the `kinesis_streams` module is not enabled. This adds extra unnecessary complexity and can easily lead to a misconfigured cluster, resulting in terraform errors (having `kinesis_events` without `kinesis_streams` breaks things badly).

## Desired Change

- Consolidate all of the code in the above modules into one module.
- Update kinesis [streams](https://github.com/airbnb/streamalert/blob/277d3b9664441feb2b2037055c101e54fb30c676/stream_alert_cli/terraform/kinesis_streams.py#L18)/[events](https://github.com/airbnb/streamalert/blob/277d3b9664441feb2b2037055c101e54fb30c676/stream_alert_cli/terraform/kinesis_events.py#L18) terraform generation code.
- Ensure the [CLI](https://git.musta.ch/csirt/streamalert/blob/1177e194fdd3bc0453dcc17dacc1e88e0e7fc307/stream_alert_cli/kinesis/handler.py#L23) handler for kinesis is working/compliant with above changes.
- Update unit tests.

Contributor guide

Open the contributing guide

Research direction

Start with terraform/modules/tf_stream_alert_kinesis_events and terraform/modules/tf_stream_alert_kinesis_streams, then inspect stream_alert_cli/terraform/kinesis_streams.py, kinesis_events.py, and kinesis/handler.py. Update the related unit tests after consolidating the modules. Done means the Terraform generation and CLI handler work with one module and the separate-module misconfiguration is no longer possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, terraform
Domain
cli, cloud, infrastructure, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.