Self hosting on AWS - Nomad DNS resolution issue during make plan

Self hosting on AWS - Nomad DNS resolution issue during make plan

Description

I am trying to self-host the application on AWS.

My current DNS setup:

• Root domain DNS is managed in DNS Made Easy.
• I commented out the original domain.tf DNS configuration.
• I updated domain.tf to only manage ACM certificate creation and validation.

Current domain.tf:

resource "aws_acm_certificate" "wildcard" {
  domain_name = var.domain_name

  subject_alternative_names = [
    "*.${var.domain_name}"
  ]

  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_acm_certificate_validation" "wildcard" {
  certificate_arn = aws_acm_certificate.wildcard.arn
}

output "acm_validation_records" {
  value = {
    for dvo in aws_acm_certificate.wildcard.domain_validation_options :
    dvo.domain_name => {
      name  = dvo.resource_record_name
      type  = dvo.resource_record_type
      value = dvo.resource_record_value
    }
  }
}

output "alb_dns_name" {
  value = aws_lb.ingress.dns_name
}

make plan

during Step 10, script fails with an external program execution error.

Error

Error: External Program Execution Failed

with module.nomad.module.template_manager.data.external.template_manager_count,
on ../modules/job-template-manager/main.tf line 5, in data "external" "template_manager_count":

5: program = ["bash", "${path.module}/scripts/get-nomad-job-count.sh"]

The data source received an unexpected error while attempting to execute the program.

The program was executed, however it returned no additional error messaging.

Program: /usr/bin/bash
State: exit status 6

Additional details

The failing Terraform data source executes:

modules/job-template-manager/scripts/get-nomad-job-count.sh

The script attempts to call the Nomad API.

When running the script manually, I receive:

curl: (6) Could not resolve host: nomad.<my-domain>

It appears script is trying to query the Nomad endpoint during terraform plan, but the DNS record for the Nomad endpoint does not exist yet or is not resolvable.

Current DNS / ACM setup

• DNS provider: DNS Made Easy(Root domain)

• Current Domain name doesn't have routing

• Wildcard certificate is configured for:

  • domain.com
  • *.domain.com

Terraform outputs the ACM validation records:

output "acm_validation_records"

but DNS records are not managed by Terraform because the DNS Made Easy provider is not configured.

Questions

1. What DNS records need to be created manually in DNS Made Easy before running make plan?

2. Should I manually create records like:

nomad.<domain>
api.<domain>
*.domain

before make plan execution? If so what can the A record for those, and I believe ALB will be provisioned after this make apply which will get the DOMAIN_NAME whichever gave in env file

3. Is get-nomad-job-count.sh expected to require an already running and reachable Nomad endpoint during terraform plan?

4. Should the Nomad job count lookup be changed to run only during terraform apply instead of terraform plan?

5. Is there an AWS-native autoscaling configuration expected for the client node pool? There is not autoscaling polices for any of the ASGs.

Any guidance on the recommended deployment order or DNS requirements would be appreciated.

Start with modules/job-template-manager/main.tf and scripts/get-nomad-job-count.sh, then run make plan to reproduce the external program failure. Trace how the Nomad hostname is formed and queried, along with the Terraform resources that provide the endpoint. Done means the deployment behavior and required DNS or execution-order change are agreed and make plan completes successfully.