How do I add a module to my Reference Architecture? #670

iancoralogix · 2023-02-24T18:00:14Z

iancoralogix
Feb 24, 2023

I have a Reference Architecture that only uses the modules it was deployed with. However, there are many other Gruntwork, open-source, and custom modules I also need to deploy. Do you know how I can do that?

Tracked in ticket #109931

Answered by iancoralogix

Feb 24, 2023

Deploy a new module in the Reference Architecture

There are several ways you can accomplish this. I'll start from the most basic form, then extend that basic structure to be fully "DRY" by following the _envcommon pattern used here at Gruntwork.

TL;DR

We must create a new terragrunt.hcl file that points to the location of the Terraform module we want to deploy. This new Terragrunt file will live in its own directory, separate from the other modules inside your Reference Architecture's repository.

📍 Writing and updating the infrastructure-live codebase is the only work that should be done on your local machine; the deployment happens through GitOps and a serverless CI/CD architecture called

View full answer

iancoralogix · 2023-02-24T21:08:54Z

iancoralogix
Feb 24, 2023
Author

Deploy a new module in the Reference Architecture

There are several ways you can accomplish this. I'll start from the most basic form, then extend that basic structure to be fully "DRY" by following the _envcommon pattern used here at Gruntwork.

TL;DR

We must create a new terragrunt.hcl file that points to the location of the Terraform module we want to deploy. This new Terragrunt file will live in its own directory, separate from the other modules inside your Reference Architecture's repository.

📍 Writing and updating the infrastructure-live codebase is the only work that should be done on your local machine; the deployment happens through GitOps and a serverless CI/CD architecture called Gruntwork Pipelines that was built for you as part of the Reference Architecture deployment.

Where we create this new directory matters. The filesystem path dictates where in AWS the infrastructure will be deployed, e.g., development/us-east-2/development/services/eks-cluster/terragrunt.hcl will look up the EKS Terraform module and then deploy it in:

The development AWS account
- In the us-east-2 AWS region
  - Into the app VPC created during the initial Reference Architecture deployment, which is represented by the second development directory in the full module path.

🌟 There are actually two VPCs in each AWS account when the Reference Architecture is initially deployed; app, where the sample app runs, and mgmt, which runs Gruntwork Pipelines. The code for both lives inside your infrastructure-live repository, and the latter is represented by something like development/us-east-2/mgmt/networking/vpc.

Since each deployed instance of Terraform module will have its own directory, you should give meaningful names to them like payments-eks-cluster or risk-mgmt-aurora-database that will be used across all environments, e.g., the data-team-ecs module be called three times, once for each environment, so it will exist in:

prod/us-west-2/data-team-ecs
stage/us-west-2/data-team-ecs
dev/us-east-2/data-team-ecs

Once you've created your module directory and terragrunt.hcl, you can open up a pull request with your changes, merge it, and Gruntwork Pipelines will automatically deploy your Terraform module into the correct account and region.

# Create a directory and file in your Reference Architecture, something like:
# development/us-east-2/development/services/demo-eks-cluster/terragrunt.hcl

# Define the Terraform module you want to deploy
terraform {
  source = "[email protected]:example-org/your-modules-here.git//modules/eks-cluster?ref=v0.0.1"
}

# Include the `terragrunt.hcl` located in the root of our Reference Architecture's
# repository because this file contains all the templating required to make our
# Terraform state and AWS provider DRY.
include "root" {
  path = find_in_parent_folders()
}

# Your module will expect some inputs. This is how to pass in values for those inputs:
inputs = {
  name   = "development-eks-cluster"
  vpc_id = "vpc-85556dc"
}

⚠️ This is not DRY, as we'd end up with duplicate code across environments. How to DRY up this code using Terragrunt and the _envcommon pattern is explained in the following guide.

Background

If we want to follow the already deployed modules' pattern, we'll first need a new directory and a terragrunt.hcl file. The directory name should be meaningful. For example, suppose we had a development environment with five different ECS clusters. In that case, we'd need a directory for each ECS cluster, as those folders contain the terragrunt.hcl file that calls the Terraform module to create them. So it would look something like this:

development/us-east-2/development/services 0 $ tree
.
├── team-a-ecs
│   └── terragrunt.hcl
├── team-b-ecs
│   └── terragrunt.hcl
├── team-c-ecs
│   └── terragrunt.hcl
├── team-d-ecs
│   └── terragrunt.hcl
└── team-e-ecs
    └── terragrunt.hcl

Each of those terragrunt.hcl files would look something like this:

# development/us-east-2/development/services/team-e-ecs/terragrunt.hcl

# This is where we tell Terragrunt what Terraform module we want to deploy.
terraform {
  source = "[email protected]:gruntwork-io/terraform-aws-service-catalog.git//modules/services/ecs-cluster?ref=v0.0.1"
}

# This is where we provide the input values to the Terraform module we want to deploy.
inputs = {
  cluster_name          = "Team E"
  cluster_instance_type = "t2.micro"
}

How to add a module using a new `terragrunt.hcl` file

In my case, I will add a demo-vpc directory with a terragrunt.hcl file that deploys an elementary module to my existing Reference Architecture. The first step would be creating my directory and Terragrunt file:

mkdir development/us-east-2/development/services/demo-vpc

cd development/us-east-2/development/services/demo-vpc && touch terragrunt.hcl

Since my demo module doesn't expect any inputs, I could quickly (but don't) get started by only having this one block in my terragrunt.hcl file:

# development/us-east-2/development/services/demo-vpc/terragrunt.hcl

terraform {
  source = "[email protected]:gruntwork-io/terraform-fake-modules.git//modules/aws/vpc?ref=main"
}

And it will run just fine, but can you spot the bug 🐞 in our thinking?

⏰ Hint: Consider the lack of Terraform configuration in our terragrunt.hcl file and remember Terragrunt is a thin wrapper for Terraform, not a replacement, and does not remove the need for fundamental concepts like providers and state.

⚠️ The issue with the above Terragrunt configuration (and not being DRY enough)

If we run this job, everything will be created as expected, but if we re-run this job, it will create duplicate resources. This is bad for a number of reasons; not only will we have duplicate infrastructure that could potentially spin up and overload other services, but we also cannot quickly and cleanly destroy the duplicate infrastructure that's been created. What we need to do is define a Terraform state to keep track of all the infrastructure that has come in and out of service. If our Terraform configuration was set up correctly, instead of recreating these resources, it would detect they already exist and that there are no configuration changes, so the deployment will do nothing. Let's understand what happened with this example and then remediate it in the following sections.

So what happened when we re-ran our deployment, and how do we fix it?

We didn't define an S3 backend to store our Terraform state in. This behavior occurs because Gruntwork Pipelines uses serverless runners on ECS for the actual deployment. When we did the first apply, the state was stored locally. However, that runner was spun down, and that local state was lost. The second apply recreated those resources, unaware of previous CI/CD jobs. The Gruntwork Pipelines architecture has many benefits but requires following best practices. Using a remote Terraform state is a major one of those best practices we enforce, but we also follow a pattern (described below) to make it painless to manage at scale.

📍 No matter what you use for CI/CD, always use a remote backend for your Terraform state!

In our demo-vpc/terragrunt.hcl file above, we got away with a lot, like Gruntwork Pipelines finding a default AWS provider automatically, but we can do better than that.

To solve this, we could (but don't) modify our demo-vpc/terragrunt.hcl to look something like this:

# development/us-east-2/development/services/demo-vpc/terragrunt.hcl

terraform {
  source = "[email protected]:gruntwork-io/terraform-fake-modules.git//modules/aws/vpc?ref=main"
}

# Non-DRY example, do not use!
 remote_state {
    backend = "s3"
    config = {
      bucket = "mybucket"
      key    = "path/to/my/key"
      region = "us-east-1"
    }
  }

But if we had thousands of resources to manage, defining that remote_state block for every Terraform module we use would become burdensome and dangerous.

🍏 Managing our Terraform state using DRY principals

In the root of our infrastructure-live repository that contains our Reference Architecture, there is a terragrunt.hcl file.

Inside this file, you'll see a block of HCL that looks like this:

# ----------------------------------------------------------------------------------------------------------------
# GENERATED REMOTE STATE BLOCK
# ----------------------------------------------------------------------------------------------------------------
# Generate the Terraform remote state block for storing state in S3
remote_state {
  backend = "s3"
  config = {
    encrypt                   = true
    bucket                    = lower("${local.name_prefix}-${local.account_name}-${local.aws_region}-tf-state")
    key                       = "${path_relative_to_include()}/terraform.tfstate"
    // ...

🆒 Note that the only hardcoded value is encrypt = true, which is desirable for this use case, but the rest are parameterized strings used to automatically gather environment context and use that to define values for things like the bucket name and terraform.tfstate location inside S3. Pretty cool, but how do we use it?

Not repeating ourselves

To access this generated remote state block that will automatically take care of everything for us, we need to include the root terragrunt.hcl into our demo-vpc/terragrunt.hcl by doing the following:

# development/us-east-2/development/services/demo-vpc/terragrunt.hcl

# Create our "VPC." In reality, this module outputs data that looks like a VPC was created).
terraform {
  source = "[email protected]:gruntwork-io/terraform-fake-modules.git//modules/aws/vpc?ref=main"
}

# Include the root `terragrunt.hcl` because this file contains all our Terraform state and AWS provider information.
include "root" {
  path = find_in_parent_folders()
}

Adding inputs before deploying our module

This part is straightforward. Even though my module doesn't expect any inputs, I will add some since 99% of modules out there expect some value.

# development/us-east-2/development/services/demo-vpc/terragrunt.hcl

# Create our "VPC." In reality, this module outputs data that looks like a VPC was created).
terraform {
  source = "[email protected]:gruntwork-io/terraform-fake-modules.git//modules/aws/vpc?ref=main"
}

# Include the root `terragrunt.hcl` because this file contains all our Terraform state and AWS provider information.
include "root" {
  path = find_in_parent_folders()
}

# This module expects no inputs, but most will expect at least a few. This is how I input those values.
inputs = {
  cidr_block = "10.222.0.0/18"
  nat_gw_count = 1
}

Merging and what to expect

Open up a pull request with your new terragrunt.hcl file and module directory, review the terragrunt plan output, approve it, and hit merge. Gruntwork Pipelines will deploy it in the correct environment automatically 🟢 . Nothing should be run from a local workstation.

Suppose this is the first time you're deploying your specific Terraform module. In that case, Terragrunt will automatically create the remote state for you based on the above changes we made to demo-vpc/terragrunt.hcl by including the root terragrunt.hcl that contains all of the generated remote state information. The scripts that initialize Gruntwork Pipelines and use git-updated-folders to understand what environments have changed are in the _ci directory.

If we look in S3, we'll see that a brand new terraform.tfstate file in a brand new location that matches our location in the repository, development/us-east-2/development/services/demo-vpc/terragrunt.hcl, in a bucket that matches lower("${local.name_prefix}-${local.account_name}-${local.aws_region}-tf-state").

Create global configurations in `_envcommon` to be DRY

We have now made the Terraform state and AWS provider information DRY, but If we wanted to launch our demo-vpc in the staging environment? The most obvious answer is to do something like this:

# staging/us-east-2/staging/services/demo-vpc/terragrunt.hcl

# Create our "VPC." In reality, this module outputs data that looks like a VPC was created).
terraform {
  source = "[email protected]:gruntwork-io/terraform-fake-modules.git//modules/aws/vpc?ref=main"  #  same as development
}

# Include the root `terragrunt.hcl` because this file contains all our Terraform state and AWS provider information.
include "root" {
  path = find_in_parent_folders()
}

# This module expects no inputs, but most will expect at least a few. This is how I input those values.
inputs = {
  cidr_block = "10.223.0.0/18"  #  different from development
  nat_gw_count = 1  #  same as development
}

Some values will be the same between development and staging, like nat_gw_count = 1. However, most values passed into Terraform modules are not unique to the environment. Therefore, we can keep this common configuration in a different HCL file and reference it from all of our terragrunt.hcl files that deploy Terraform modules.

To start the _envcommon pattern in this example, we will create a new HCL file at this location, _envcommon/services/demo-vpc.hcl, which will hold all of our global values. We can always override the values in this file, but they are a great way to keep our infrastructure-live repository DRY:

# _envcommon/services/demo-vpc.hcl

terraform {
  # We can override this in our terragrunt.hcl files. This is useful for promoting changes across environments.
  source = "${local.source_base_url}?ref=main"
}

locals {
  source_base_url = "[email protected]:gruntwork-io/terraform-fake-modules.git//modules/aws/vpc"
}

inputs = {
  # Safe default values available to all environments are defined here. They can be overridden in our terragrunt.hcl files.
  cidr_block   = "10.0.0.0/16"
  nat_gw_count = 1
}

How to use the global configuration in `_envcommon`

We can now use a very generic terragrunt.hcl whenever we need to call this demo-vpc in our environments:

# development/us-east-2/development/services/demo-vpc/terragrunt.hcl

# Create our "VPC." In reality, this module outputs data that looks like a VPC was created).
terraform {
  source = "${include.envcommon.locals.source_base_url}?ref=main"
}

# Follow the _envcommon pattern and use the configuration located at _envcommon/services/demo-vpc.hcl
include "envcommon" {
  path = "${dirname(find_in_parent_folders())}/_envcommon/services/demo-vpc.hcl"
  # We want to reference the variables from the included config in this configuration, so we expose it.
  expose = true
}

# Include the root `terragrunt.hcl` because this file contains all our Terraform state and AWS provider information.
include "root" {
  path = find_in_parent_folders()
}

inputs = {
  cidr_block = "10.222.0.0/18"  # Unique only to development, so we override the default _envcommon value of 10.0.0.0/16.
}

Using prior art to maximize `_envcommon` DRYness for new modules

Your Reference Architecture was deployed with several dozen modules. A good one to look at would be _envcommon/data-stores/redis.hcl to see dependency blocks. The purpose of dependency blocks is to extract values from other modules' existing infrastructure that your new module depends on, e.g., your Redis cluster needs a VPC to run in, and the Redis module expects a value for vpc_id.

# ---------------------------------------------------------------------------------------------------------------------
# Dependencies are modules that need to be deployed before this one.
# ---------------------------------------------------------------------------------------------------------------------

dependency "vpc" {
  config_path = "${get_terragrunt_dir()}/../../networking/vpc"

  mock_outputs = {
    // ...
  }
  mock_outputs_allowed_terraform_commands = ["validate", ]
}

dependency "network_bastion" {
  config_path = "${get_terragrunt_dir()}/../../networking/openvpn-server"
  // ...

Many AWS resources depend on a VPC, and many Terraform modules expect a value for vpc_id to ensure the resources are deployed to the correct network location. We can be very DRY here. Rather than hardcoding the vpc_id in a .tfvars file or doing a remote_state data call, we can use the filesystem and directory structure to access values output from other modules using config_path = // ...

We access the dependency address space to pass values into our inputs = {} for the Terraform module we want to deploy. In this example, there are many safe default values like instance_type = "cache.t3.micro" to ensure we don't accidentally launch an expensive instance type, but we also use values from three different Terraform modules that the Redis depends on:

# ---------------------------------------------------------------------------------------------------------------------
# MODULE PARAMETERS
# These are the variables we must pass in to use the module specified in the terragrunt configuration above.
# This defines the parameters that are common across all environments.
# ---------------------------------------------------------------------------------------------------------------------
inputs = {
  # Redis cluster name must be < 40 characters
  name = substr("redis-${local.name_prefix}-${lower(local.account_name)}", 0, 40)

  instance_type = "cache.t3.micro"
  vpc_id        = dependency.vpc.outputs.vpc_id
  subnet_ids    = dependency.vpc.outputs.private_persistence_subnet_ids
  redis_version = "5.0.6"

  replication_group_size    = 1
  enable_multi_az           = false
  enable_automatic_failover = false
  parameter_group_name      = "default.redis5.0"
  enable_cloudwatch_alarms  = true
  alarms_sns_topic_arns     = [dependency.sns.outputs.topic_arn]

  # Here we allow any connection from the private app subnet tier of the VPC. You can further restrict network access by
  # security groups for better defense in depth.
  allow_connections_from_cidr_blocks     = dependency.vpc.outputs.private_app_subnet_cidr_blocks
  allow_connections_from_security_groups = [dependency.network_bastion.outputs.security_group_id]

  # Only apply changes during the scheduled maintenance window, as certain DB changes cause degraded performance or
  # downtime. For more info, see: https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/Clusters.Modify.html
  # We default to false, but in non-prod environments we set it to true to immediately roll out the changes.
  apply_immediately = false
}

At the bottom of _envcommon/data-stores/redis.hcl we can see those dependencies in action:

inputs = {
  // ...
  vpc_id        = dependency.vpc.outputs.vpc_id
  subnet_ids    = dependency.vpc.outputs.private_persistence_subnet_ids

  // ...
  alarms_sns_topic_arns     = [dependency.sns.outputs.topic_arn]

  // ...
  allow_connections_from_cidr_blocks     = dependency.vpc.outputs.private_app_subnet_cidr_blocks
  allow_connections_from_security_groups = [dependency.network_bastion.outputs.security_group_id]

}

Wrapping up

We have successfully made our configuration DRY in the following areas:

Terraform state
Terraform providers
Common Terraform module configuration
Passing of outputs from one module to the inputs of another module

And additionally, because of Terragrunt, we've unlocked additional benefits:

Created a path for developers to copy-paste terragrunt.hcl files with safe defaults to self-serve infrastructure.
Used GitOps to create new infrastructure.
Created a pattern to merge different versions of the same module into different environments.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gruntwork

How do I add a module to my Reference Architecture? #670

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Gruntwork

How do I add a module to my Reference Architecture? #670

iancoralogix Feb 24, 2023

Deploy a new module in the Reference Architecture

TL;DR

Replies: 1 comment

iancoralogix Feb 24, 2023 Author

Deploy a new module in the Reference Architecture

TL;DR

Background

How to add a module using a new terragrunt.hcl file

⚠️ The issue with the above Terragrunt configuration (and not being DRY enough)

So what happened when we re-ran our deployment, and how do we fix it?

🍏 Managing our Terraform state using DRY principals

Not repeating ourselves

Adding inputs before deploying our module

Merging and what to expect

Create global configurations in _envcommon to be DRY

How to use the global configuration in _envcommon

Using prior art to maximize _envcommon DRYness for new modules

Wrapping up

iancoralogix
Feb 24, 2023

iancoralogix
Feb 24, 2023
Author

How to add a module using a new `terragrunt.hcl` file

Create global configurations in `_envcommon` to be DRY

How to use the global configuration in `_envcommon`

Using prior art to maximize `_envcommon` DRYness for new modules