airbnb / airbnb/streamalert

Deploying multiple clusters to different regions

Open
#418 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

## Background

When deploying multiple StreamAlert clusters configured for different AWS regions, all the clusters are created in the default aws region.

This is because the terraform modules do not correctly specify which aws provider (and region)
to use depending on the configured cluster. According to the [terraform docs](https://www.terraform.io/docs/configuration/providers.html#multiple-provider-instances), the correct way to do this is to create a new `provider` directive with an alias you can reference in each `resource`. For example:

```
provider "aws" {
alias = "${var.cluster}"
region = "${var.region}"
}

resource "aws_kinesis_stream" "stream_alert_stream" {
provider = "aws.${var.cluster}"
name = "${var.stream_name}"
shard_count = "${var.shards}"
retention_period = "${var.retention}"
}
```

This will create the resource in the correct region, but requires a `provider` argument to each `resource`. Additionally the cluster and region need to be passed into each terraform module.

Unfortunately if you removed a module from an existing configuration, terraform will use the default provider to attempt to destroy it. This is because the module defined provider won't exist anymore. Also, strangely, a child module will use the provider of the parent terraform file over it's own.

Interesting relevant discussions:
https://github.com/hashicorp/terraform/issues/15961
https://github.com/hashicorp/terraform/issues/15762

## Desired Change

Running `python manage.py terraform build|init` should correctly deploy clusters to their configured regions.

Looks like terraform may be improving the way they handle providers with modules. Not sure how to best fix this.

Contributor guide

Open the contributing guide

Research direction

Start with the terraform path invoked by `python manage.py terraform build|init`, then trace how cluster and region settings reach the Terraform modules and providers. Confirm the behavior with a configuration containing clusters in different AWS regions; done means each cluster and its resources deploy to its configured region without breaking module removal.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, terraform
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.