Deploying multiple clusters to different regions
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
When deploying multiple StreamAlert clusters configured for different AWS regions, all the clusters are created in the default aws region.
This is because the terraform modules do not correctly specify which aws provider (and region)
to use depending on the configured cluster. According to the [terraform docs](https://www.terraform.io/docs/configuration/providers.html#multiple-provider-instances), the correct way to do this is to create a new `provider` directive with an alias you can reference in each `resource`. For example:
```
provider "aws" {
alias = "${var.cluster}"
region = "${var.region}"
}
resource "aws_kinesis_stream" "stream_alert_stream" {
provider = "aws.${var.cluster}"
name = "${var.stream_name}"
shard_count = "${var.shards}"
retention_period = "${var.retention}"
}
```
This will create the resource in the correct region, but requires a `provider` argument to each `resource`. Additionally the cluster and region need to be passed into each terraform module.
Unfortunately if you removed a module from an existing configuration, terraform will use the default provider to attempt to destroy it. This is because the module defined provider won't exist anymore. Also, strangely, a child module will use the provider of the parent terraform file over it's own.
Interesting relevant discussions:
https://github.com/hashicorp/terraform/issues/15961
https://github.com/hashicorp/terraform/issues/15762
## Desired Change
Running `python manage.py terraform build|init` should correctly deploy clusters to their configured regions.
Looks like terraform may be improving the way they handle providers with modules. Not sure how to best fix this.
Contributor guide
Research direction
Start with the terraform path invoked by `python manage.py terraform build|init`, then trace how cluster and region settings reach the Terraform modules and providers. Confirm the behavior with a configuration containing clusters in different AWS regions; done means each cluster and its resources deploy to its configured region without breaking module removal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, terraform
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100